You don’t say what denomination James Kennedy was. If he was RC then sadly the RC baptism records for Inver parish only start in 1861. If Church of Ireland they start in 1805 and if Presbyterian, then Donegal 1st appears to be the nearest church and their records start in 1824. So unless he was Church of Ireland, there may not be any trace of his birth. (The Church of Ireland records have been copied and are in PRONI, the public record office in Belfast.).
You say that James married Mary Byrne and assume it was in Tullylish RC church in Co Down in 1848. That’s perfectly possible but it’s 130 miles from Inver to Co Down, and not somewhere Donegal people often married. Most people married someone local to where they lived, unless they happened to be in a profession that involved travel eg a mason, soldier or suchlike. Unless you have information linking James to Tullylish, I’d suspect that he is more likely to have married in Inver or close by in Co. Donegal but the records don’t exist which is why you haven’t found a match there.
Here’s a link to a site with a page on genealogy in Inver, which you may find helpful: