Initially I tried to use Segger J-Link to detect the processor, but without luck:

J-Link>con
Device "LPC2378" selected.


Connecting to target via JTAG
TotalIRLen = 4, IRPrint = 0x01
Failed to identify target. Resetting via Reset pin and trying again.
TotalIRLen = 4, IRPrint = 0x01
TotalIRLen = 4, IRPrint = 0x01
Failed to identify target. Resetting via Reset pin and trying again.
TotalIRLen = 4, IRPrint = 0x01
Cannot connect to target.
J-Link>

I started to suspect that I bought a damage board.

Then I found a comment saying that the board needs 9V power supply:
https://www.embeddedrelated.com/showthread/lpc2000/41656-1.php

But that is strange because I was powering the board using USB cable and the PWR_SEL jumper correctly selected to position 3.

Anyway I found a 9V power-supply and used that, but again JLink failed to find it.

Then I decide to test using OpenOCD with a Versaloon-Link programmer that I have here and it worked like a charm:

$ sudo openocd -f interface/vsllink.cfg -f target/lpc2378.cfg
[sudo] password for alan:
Open On-Chip Debugger 0.11.0
Licensed under GNU GPL v2
For bug reports, read
http://openocd.org/doc/doxygen/bugs.html
init_targets
Info : Listening on port 6666 for tcl connections
Info : Listening on port 4444 for telnet connections
Warning - assuming default core clock 4MHz! Flashing may fail if actual core clock is different.
Info : auto-selecting first available session transport "jtag". To override use 'transport select '.
Info : Versaloon(0x15)by Simon(compiled on Jul 8 2011)
Info : USB_TO_XXX abilities: 0x0000076E:0x010001EF:0xC0000007
Info : clock speed 500 kHz
Info : JTAG tap: lpc2378.cpu tap/device found: 0x4f1f0f0f (mfg: 0x787 (), part: 0xf1f0, ver: 0x4)
Info : Embedded ICE version 7
Error: EmbeddedICE v7 handling might be broken
Info : lpc2378.cpu: hardware has 2 breakpoint/watchpoint units
Info : starting gdb server for lpc2378.cpu on 3333
Info : Listening on port 3333 for gdb connections

So next time you face some issue with JLink, try to use OpenOCD!