JSON logging

UnrealIRCd 6 introduces optional JSON logging in log files and in server notices sent to snomasks (JSON over IRC). This makes it easy to parse and process log messages by programs and bots.

The JSON log also contains a lot more detail than the text log because it includes, for each user, things like realname, account, operlogin, geoip information, tls information, channels, etc.

If you are a fan of JSON, then you may also be interested in JSON-RPC which allows remotely controlling UnrealIRCd. In UnrealIRCd 6.1.0 and later JSON-RPC also allows real-time streaming of log events over websockets with the log.subscribe API call.

JSON Example output

This is an example of the JSON log output when someone types JOIN #five:

Enabling JSON logging

JSON logs can be written to disk, received over IRC or received over websockets via JSON-RPC.

Enabling in disk logging

Usually you will want to keep your ircd.log logging as plaintext (non-JSON) because it is easy to read by humans.

So, just add another NEW log block with JSON logging enabled, to for example ircd.json.log, like this:

log {
        source {
                all;
                !debug;
                !join.LOCAL_CLIENT_JOIN;
                !join.REMOTE_CLIENT_JOIN;
                !part.LOCAL_CLIENT_PART;
                !part.REMOTE_CLIENT_PART;
                !kick.LOCAL_CLIENT_KICK;
                !kick.REMOTE_CLIENT_KICK;
        }
        destination {
                file "ircd.json.log" { maxsize 250M; type json; }
        }
}

As you can see, you simply set the log::destination::file::type to json.

And then /REHASH.

Colorized JSON

The JSON logged to the file is not nicely formatted and does not contain color. The colorized JSON log as shown in the screenshot on this page is generated with the tool jq. So this is all completely optional, just mentioning it for anyone interested in seeing it colorized:

1. Install jq, eg through: sudo apt-get install jq
2. Then tail (or cat) the log through it, eg: tail -F ~/unrealircd/logs/ircd.json.log|jq

Enabling on IRC

IRCOps receive server notices through snomasks. It is possible to receive the associated JSON logging data on IRC as well.

No IRC client supports this natively out of the box, so here is a technical explanation:

1. Send CAP REQ unrealircd.org/json-log to indicate that you want to receive JSON logging, eg /QUOTE CAP REQ unrealircd.org/json-log
2. Become IRCOp using /OPER (if you are not one already)
3. Open up a debug window in your client so you can see raw IRC traffic. Eg: /debug -pt @debug in mIRC. Other clients typically have something similar.
4. Now when you receive a server notice via snomasks (for example a new user connecting), then the JSON data will be sent in the unrealircd.org/json-log message tag.

In mIRC scripting the JSON data in the server notice will be in $msgtags(unrealircd.org/json-log)

Your bot/program can then parse this as JSON data and process it. The format is much more consistent than regular text messages and typically shows a lot more details about the objects involved (client, channel, etc)

NOTE: If you use ZNC or another bouncer, then this won't work because it does not pass on the message tag.

Example on IRC

First, request the capability, oper up, and for this example we will also set the join snomask (+s +j):

CAP REQ :unrealircd.org/json-log
OPER ...........
MODE Yournick +s +j

Then when another client types JOIN #five we will receive the following raw server traffic:

@unrealircd.org/json-log={"timestamp":"2022-05-23T11:02:13.519Z","level":"info","subsystem":"join","event_id":"LOCAL_CLIENT_JOIN","log_source":"maintest.test.net","msg":"User\sSyzop\sjoined\s#five","client":{"name":"Syzop","id":"001F9RV02","hostname":"xyz.example.org","ip":"198.51.100.1","server_port":6697,"client_port":37030,"details":"[email protected]","connected_since":"2022-05-23T11:02:06.000Z","idle_since":"2022-05-23T11:02:06.000Z","user":{"username":"~x","realname":"IRC\sUser","vhost":"oper/netadmin.test.net","cloakedhost":"Mask-8608861.example.net","servername":"maintest.test.net","reputation":12,"security-groups":["unknown-users","tls-users"],"modes":"iotxz","geoip":{"country_code":"NL"},"snomasks":"bcdfkoqsBOS","operlogin":"Syzop","operclass":"netadmin-with-override","channels":["#five","#four","#three","#two","#one"]}},"tls":{"certfp":"aafe66a7d808e1fca077805c54b1274a92d30c3023e35ec130f358d238218296","cipher":"TLSv1.3-TLS_CHACHA20_POLY1305_SHA256"},"channel":{"name":"#five","creation_time":"2022-05-23T11:02:13.000Z","num_users":1,"modes":"ntf\s[4j#R1,5m#M1,3n#N1,3t#b1]:2"},"modes":"o","source":{"file":"join.c","line":292,"function":"_join_channel"}}
:maintest.test.net NOTICE oper :join.LOCAL_CLIENT_JOIN [info] User Syzop joined #five

With the help of a JSON library the client can then decode it to objects.

Enabling in JSON-RPC

In UnrealIRCd 6.1.0 and later JSON-RPC allows real-time streaming of log events over websockets with the log.subscribe API call.

UnrealIRCd JSON data

Every JSON log message contains at least the following data:

Variable	Description	Example value
timestamp	Date/time when the log message was generated	2022-05-23T11:02:06.000Z
level	Severity of the log message	info
subsystem	Subsystem which generated the log message	join
event_id	Event ID of the log message	LOCAL_CLIENT_JOIN
log_source	Servername which generated the log message. This is because UnrealIRCd will receive various log messages from remote servers.	irc1.example.net
msg	The human readable message (this is used in plaintext logs and snomasks).	User Syzop joined #five

In addition to that there are a couple of fields that are often included (but not always!):

Variable	Description	Example value
client	The client causing the event. For example, for a KICK this would be the person issuing the /KICK command.	See client object
target	The client being the victim of the event. For example, for a KICK this would be the victim.	See client object
channel	The channel where the event is taking place.	See channel object

There are many possible log messages and the additional fields that are available for each event differ. You can look at the List of all log messages for a particular event to see how it is generated. Or you can capture real log items or IRC data.

client object

For JSON logging we use a Detail level of 3 for client objects.

This can be a user, a server or a freshly accepted connection that is still in the handshake and we don't know yet what it will be.

Detail level	Variable	Description	Example value
0+	name	The name of the client (nick name, server name, ..)	Syzop
0+	id	The unique client ID	001F9RV02
1+	hostname	Resolved hostname (or IP)	xyz.example.org
1+	ip	IP address of the user (empty for services)	198.51.100.1
1+	details	Detailed description of the client. The output depends on the type of client and what information we have.	nick!user@host [email protected] [198.51.100.1]
1+	geoip	GeoIP information regarding the IP address of the user (not always available)	(see next)
1+	geoip.country_code	The ISO 3166-1 alpha-2 country code	NL
1+	geoip.asn	The AS (Autonomous System) number that the user is connecting from	16276
1+	geoip.asname	The AS (Autonomous System) name that the user is connecting from	OVH SAS
2+	server_port	Server TCP port (unrealircd side)	6697
2+	client_port	Client TCP port (client side)	37030
2+	connected_since	Date/time when the client connected (only available for local clients)	2022-05-23T11:02:06.000Z
2+	idle_since	Last time the client said anything in PM or channel (only available for local clients)	2022-05-23T11:02:06.000Z
2+	tls	If the client is using SSL/TLS then this contains a tls object which has two members	(see next)
2+	tls.cipher	The TLS cipher negotiated with the client	TLSv1.3-TLS_CHACHA20_POLY1305_SHA256
2+	tls.certfp	The client Certificate fingerprint (if any)	aafe66a7d808e1fca077805c54b1274a92d30c3023e35ec130f358d238218296
2+	user	If the client is a user (a person) then additional information is available in this object.	See client.user object
2+	server	If the client is a server then additional information is available in this object.	See client.server object

See also below for the user and server objects.

client.user object

If the client is a user then there is a user object within the client object:

Detail level	Variable	Description	Example value
2+	user.username	The user name / ident	~xyz
2+	user.realname	The real name / GECOS	IRC User
2+	user.vhost	If the user is +x or +t this contains the vhost/cloakedhost	oper/netadmin.test.net
2+	user.cloakedhost	The calculated cloaked host, even if the user is not +x	Mask-8608861.example.net
2+	user.servername	The server to which the user is connected	irc.example.net
2+	user.account	The account name, if the user is logged in to Services.	SomeAccount
2+	user.reputation	The reputation score.	10000
2+	user.security-groups	The security groups that the user is in.	["known-users"]
2+	user.modes	The user modes of the user.	"iwx"
2+	user.snomasks	The snomasks of the user, if they are IRCOp.	"iwx"
2+	user.operlogin	The name of the oper { } block, if they are IRCOp.	"iwx"
2+	user.snomasks	The oper::operclass, if they are IRCOp.	"iwx"
3	user.channels	The channels the user is in. Minimized and capped at 384 bytes total length.	["#a","#b","#c"]
4	user.channels	The channels the user is in with the level	[{"name": "#a", "level": "o"}, {"name": "#b", "level": "o"}, {"name": "#c", "level": "o"}]

client.server object

If the client is a server then there is a server object within the client object:

Detail level	Description	Example value
2+	server.info	The server description	Primary IRC server of example.net
2+	server.num_users	Number of currently connected users	123
2+	server.boot_time	Date/time the server was started	2022-05-23T11:02:06.000Z
2+	server.synced	Set to 1 if the server is synced, 0 if the server is currently linking in and syncing users/channels/etc.	1
2+	server.features.software	Server software version in use	UnrealIRCd 6.0.0
2+	server.features.protocol	UnrealIRCd protocol version	6000
2+	server.features.usermodes	User modes supported by this server	abcd
2+	server.features.chanmodes	Channel modes supported by this server. This is an array of 4 strings, the same format as IRC numeric 005 CHANMODES=.	['abc','def','ghi','jkl']
2+	server.features.nick_character_sets	Nick Character Sets supported by this server. Comma separated string.

channel object

For JSON logging we use a Detail level of 1 for channel objects. Level 2 and higher are only used by JSON-RPC.

Detail level	Variable	Description	Example value
0+	name	The name of the channel	#five
1+	creation_time	Date/time the channel was first created	2022-05-23T11:02:13.000Z
1+	num_users	Number of users in the channel	1
1+	topic	Topic of the channel (if a topic is set)	Welcome everyone
1+	topic_set_by	Name of the user who set the topic (if a topic is set)	Syzop
1+	topic_set_at	Date/time when the topic was set (if a topic is set)	2022-05-23T1:30:00.000Z
1+	modes	Channel modes that are set (not including list modes)	ntf [4j#R1,5m#M1,3n#N1,3t#b1]:2
2+	bans	List of all bans (+b) in the channel	[{"name": "some!nice@ban","set_by": "some_user","set_at": "2023-01-04T07:52:54.000Z"}]
2+	ban_exemptions	List of all ban exceptions (+e) in the channel	[{"name": "some!nice@exempt","set_by": "some_user","set_at": "2023-01-04T07:52:54.000Z"}]
2+	invite_exceptions	List of all invite exceptions (+I) in the channel	[{"name": "some!nice@invex","set_by": "some_user","set_at": "2023-01-04T07:52:54.000Z"}]
3	members	List of members (nicks) that are in the channel - simple	[{"level": "o", "name": "one", "id": "001QMQ00B"}, {"name": "two", "id": "001TKVY0A"}]
4	members	List of members (nicks) that are in the channel - more details	[{"level": "o", "name": "one", "id": "001QMQ00B", "hostname": "localhost", "ip": "127.0.0.1", "details": "one!test@localhost"}, {"name": "two", "id": "001TKVY0A", "hostname": "localhost", "ip": "127.0.0.1", "details": "two!~x@localhost"}]
5	members	List of members (nicks) that are in the channel - almost completely expanded. Only the members.user.channels object is NOT added. That field would otherwise show all the channels the user is in, which is generally not useful if you are using channel.* API calls.	Too big to show here
6	members	Don't use this level, it's for JSON logging only.	Don't use this level
7	members	List of members (nicks) that are in the channel - completely expanded, not recommended.	Too big to show here

socket_error object

Included in socket errors. Such as a socket being closed due to a network problem or otherwise.

Variable	Description	Example value
error_code	The error code as an integer	5
error_string	The error as a string	Input/output error

NOTE: The error codes and strings come directly from the OS and C library, hence they will differ between OS's (Linux, Windows, FreeBSD, ..).

tls_error object

Included in SSL/TLS errors. Such as failure to read a certificate file or a socket being closed due to a network problem.

Variable	Description	Example value
name	The first encountered error as a printable string, for convenience. This will be identical to error_stack[0].string.	5
error_stack	An array of items on the OpenSSL "error stack"	(see next)
error_stack[].code	The OpenSSL error code as an integer	5
error_stack[].string	OpenSSL error as a string	???

NOTE: The error codes and strings come directly from OpenSSL (outside our control).

link_block object

This is the content of a Link block. It is typically included in server linking errors, but only if it is known which server to link to/from.

Variable	Description	Example value
name	Name of the link block, as in link irc1.example.net { }	irc1.example.net
hostname	link::outgoing::hostname for outgoing connects (may not be present)	irc1.example.net
hostname	Resolved IP for outgoing connects (may not be present)	198.51.100.1
file	link::outgoing::file for outgoing connects (not often used, likely not present)	/tmp/server.sock
port	link::outgoing::port for outgoing connects (may not be present)	6900
bind-ip	link::bind-ip for outgoing connects	198.51.100.2
class	link::class	servers

tkl object

This is a server ban, such as a GLINE or Spamfilter. It is also used for ban exceptions (ELINE).

Variable	Description	Example value
type	Type of the server ban. One of: gline, kline, gzline, zline, spamfilter, qline, except, shun, local-qline, local-exception, local-spamfilter.	gline
type_string	Type of the server ban in a more friendly user printable string. Possibly prefixed with "Soft" eg "Soft G-Line".	G-Line
name	The target of the ban or except. For Spamfilter this is the regex/matcher.	*@*.badisp.example.net
set_by	Name of the person or server who set the ban	Syzop
set_at	Date/Time when the server ban was added	2022-05-23T11:02:06.000Z
set_at_string	Date/Time when the server ban was added in a more human printable string.	Mon May 23 11:02:06 2022
expire_at	Date/Time when the server ban will expire. NULL means: never.	2023-05-23T10:00:00.000Z
expire_at_string	Date/Time when the server ban will expire in a more human printable string. Uses "Never" for never.	Tue May 23 10:00:00 2023
duration_string	How long the ban will last from this point in time (human printable). Uses "permanent" for forever.	permanent
set_at_delta	How many seconds ago the ban was placed.	1800
reason	The reason of the ban	Lots of abuse from this ISP
exception_types	Only for ban exceptions! Specifies the exception types (letters). See HELPOP ELINE for a list of those letters.	kGzZsF
match_type	Only for spamfilters! The matching type. One of: simple, regex	regex
ban_action	Only for spamfilters! The action to take on spamfilter hit.	gline
spamfilter_targets	Only for spamfilters! Which targets the spamfilter must filter on.	cpnN