simple chat server in C with TCP
Kragen Sitaker kragen@pobox.com
Sun, 26 Dec 1999 19:10:43 -0500 (EST)
* Previous message: APLish basic stats program in Perl
* Next message: chat-server patch
* Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
This is in ar format; if you start from the line that says !<arch>, you
should be able to extract it with ar xv. Failing that, you can of
course extract it with a text editor.
!<arch>
// 18 `
chat-server.txt/
/0 946252827 1069 1106 100644 6053 `
Here's a toy chat server --- this time using TCP in C. I expect some
of the pieces will come in handy later.
event.c and event.h
-------------------
The core of it is a pretty sturdy select() loop in C, in event.c and
event.h, totaling about 100 lines. It doesn't do any dynamic
allocation at all.
The select() loop could use extending to handle signals, scheduled
events, and possibly the RT-queue SIGIO siginfo stuff for reasonable
scalability under Linux. (Its scalability as-is is no worse under
Linux than under any other OS, but Linux has a solution to the
select() problem, as seen in Zach Brown's phhttpd.) But I'm not going
to extend it unless I have a use for it (so I know I'm extending it
correctly) and I have test data for it (so I know it works).
die.c, die_test.c, and die.h
----------------------------
Error handling is taken care of by a macro called die_if, which takes
a condition and a pointer to an error message format string. The
format string is like printf's format string, but the %-sequences are
different; it supports %% (literal %), %f (the file where the error
occurred), %l (the line where the error occurred), %c (the expression
that evaluated to true to cause the failure), and %e (the string
corresponding to the current value of errno).
I had high-minded ideas that I would put all of my error messages in a
separate .c file and link that in, so translating to different
languages would be a breeze, but I haven't done that yet. There are
only six error messages at the moment.
Also, I wanted to be able to catch different errors with a macro that
used setjmp(), but I haven't needed that yet, so I haven't implemented
it. Right now die_if() just calls exit() when it's done printing its
message to stderr.
die_test.c is there to test the die_if() function.
All the die stuff is about 50 lines long.
kstr.c, kstr.h, and kstr_test.c
-------------------------------
One of the major problems with doing this sort of thing in C is
dynamically allocating strings. I wrote a relatively bare-bones
string package called 'kstr', whose architecture is based on that of
djb's stralloc (indeed, the data structure is identical); it's about
125 lines long. It could not be called complete; I only implemented
what I needed, plus what I wrote tests for, which turned out to be
'read bytes from a file descriptor' and 'append a bunch of bytes',
plus mostly internal stuff.
Despite the primitiveness of the string package, it is more than
sufficient for the chat server's needs.
kstr_test.c is a line-by-line cat program to test the basic string
operations.
talloc.c, talloc.h, and mem-used
--------------------------------
I was having trouble with memory leaks because I was creating kstrs on
the fly in the chat server and forgetting to delete them; to help me
track them down, I wrote a couple of functions to wrap malloc() and
free() and print out information about allocation and deallocation,
and a Perl script to analyze the results. These total about 50 lines;
the tracing output is currently turned off, but can be turned back on
by uncommenting a #define in talloc.c.
chat-server.c
-------------
This is the actual chat server, about 250 lines. It abuses a kstr to
store its per-client data, and uses kstrs to buffer outgoing data. It
lives in a select() loop, and normally should make no blocking system
calls other than select().
I learned some interesting things about write(2) and read(2). If you
read() a socket, and the socket has less data than you request, you
get whatever is available (and you get told how much you got). I
always thought write() worked the same way, but it turns out that (at
least on Solaris 7) it blocks until it has written all of the data you
requested, unless you set O_NDELAY or O_NONBLOCK on the socket.
I managed to get netcat and the server to deadlock several times while
testing this.
I was able to pump about 4 megabytes of NULs per second through this
server in 8K chunks on the Ultra at work. I had two clients connected
to the server; one was sending. I lost data (see the notes in the
code about this) -- about 6.3% on one client and about 20% on the
other.
Hit return when you want the server to quit. I added this feature so
I could balance the memory books and so I could get some gmon.out
output for gprof. (Evidently the INT signal doesn't make gprof write
out its stuff; you have to exit properly.) (Oh, because of the
primitive way this is handled, the server may quit even when running
in the background if you type in the foreground. Solution: don't run
it in the background.)
When running the server, you can telnet to it on port 17224 and type
messages, which get forwarded to everyone else who's telnetted to it
at that time. Obviously this is not yet a replacement for IRC.
Oh, I've heard there's a potential problem with port scans on some
OSes: the listening socket will select() as readable, but the
connection will close before you accept() on it, which makes the
accept() block until someone else connects. I haven't done anything
to prevent this (such as marking the socket non-blocking and handling
accept() errors).
I end up passing around a file descriptor all over the place; it might
be better to pass around a pointer to a client_info instead of calling
get_cip() everywhere.
Bugs
----
I had lots of bugs when I was writing this code. I forgot to adjust
the lengths of kstrs when I modified them, in kstr_append and in
new_client_conn(); I misunderstood APIs as simple as write(); I forgot
to deallocate strings; I forgot to close file descriptors; I sent the
wrong From address on messages (the destination instead of the
source); I didn't ignore SIGPIPE; I forgot to initialize all kinds of
things (which was mostly OK, since they happened to be initialized to
zero); I had an off-by-one error in allocating and zeroing one fewer
struct client_info than I needed in new_client_conn(); I left out
#includes.
As far as I know, though, all the bugs are fixed now. Most of them
were in chat-server.c.
event.c/ 946245183 1069 1106 100644 1790 `
/* event loop */
#include <sys/time.h>
#include <string.h>
#include <stdio.h>
#include "event.h"
void init_fdvec(fdvec *e)
{
FD_ZERO(&e->fds);
memset(&e->f, '\0', sizeof(e->f));
e->size = 0;
}
void init_eventset(eventset *e)
{
init_fdvec(&e->read);
init_fdvec(&e->write);
}
void on_event(fdvec *e, int fd, void (*f)(int fd))
{
FD_SET(fd,&e->fds);
e->f[fd] = f;
if (fd >= e->size) e->size = fd + 1;
}
void on_event_nop(fdvec *e, int fd)
{
int i;
FD_CLR(fd,&e->fds);
e->f[fd] = NULL;
if (fd == e->size-1) {
e->size = 0;
for (i = 0; i != fd; i++) {
if (FD_ISSET(i, &e->fds)) e->size = i + 1;
}
}
}
void handle_events(eventset *e)
{
fd_set readfds, writefds;
int maxfd;
int i;
int nothing_to_write = 1;
readfds = e->read.fds;
writefds = e->write.fds;
maxfd = (e->read.size > e->write.size) ? e->read.size : e->write.size;
select(maxfd, &readfds, &writefds, 0, 0);
/* the apparently redundant checks on e->read.fds and
e->write.fds are there in case a previous event handler has
removed the event handlers for this file descriptor since
the select() call. */
/* outgoing data has higher priority than incoming data. This
policy is manifested by not scheduling reads of incoming data
while there's outgoing data that can be scheduled. */
for (i = 0; i != maxfd; i++) {
if (FD_ISSET(i, &writefds) && FD_ISSET(i, &e->write.fds)) {
/* fprintf(stderr, "%d writable\n", i); */
e->write.f[i](i);
nothing_to_write = 0;
}
}
if (nothing_to_write) {
for (i = 0; i != maxfd; i++) {
if (FD_ISSET(i, &readfds) && FD_ISSET(i, &e->read.fds)) {
/* fprintf(stderr, "%d readable\n", i); */
e->read.f[i](i);
}
}
}
}
event.h/ 946248036 1069 1106 100644 471 `
/* event loop. */
typedef struct
{
fd_set fds;
void (*f[FD_SETSIZE])(int fd);
int size;
} fdvec;
typedef struct
{
fdvec read;
fdvec write;
} eventset;
void init_eventset(eventset *e);
/* register event handler */
void on_event(fdvec *e, int fd, void (*f)(int fd));
/* remove event handler */
void on_event_nop(fdvec *e, int fd);
/* handle pending events, or wait for an event */
void handle_events(eventset *e);
die.c/ 946227821 1069 1106 100644 991 `
#include <stdio.h>
#include <string.h>
#include <errno.h>
void die_if_func(int whether, char *cond, char *file, int line, char *msg)
{
if (whether) {
char *s;
for (s = msg; *s; ++s) {
if (*s != '%') {
putc(*s, stderr);
} else {
++s;
switch(*s) {
case '\0':
fprintf(stderr, "(Unterminated %% sequence in error string)\n");
goto done_with_msg;
case '%':
putc('%', stderr);
break;
case 'f':
fprintf(stderr, "%s", file);
break;
case 'l':
fprintf(stderr, "%d", line);
break;
case 'c':
fprintf(stderr, "%s", cond);
break;
case 'e':
fprintf(stderr, "%s", strerror(errno));
break;
default:
fprintf(stderr, "(invalid %% sequence %%%c in error string)\n", *s);
break;
}
}
}
done_with_msg:
putc('\n', stderr);
fflush(stderr); /* it's probably line-buffered anyway, but I'll do this anyhow */
exit(1);
}
}
/* standard errors */
char *out_of_memory = "Out of memory at %f:%l (says %c) (error %e)";
die_test.c/ 946227581 1069 1106 100644 74 `
#include "die.h"
int main()
{
die_if(1, out_of_memory);
return 0;
}
die.h/ 946227606 1069 1106 100644 193 `
#define die_if(cond,msg) (die_if_func(cond,#cond,__FILE__,__LINE__,msg))
void die_if_func(int whether, char *cond, char *file, int line, char *msg);
/* standard errors */
char *out_of_memory;
kstr.c/ 946240373 1069 1106 100644 1720 `
#include <string.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include "die.h"
#include "kstr.h"
#include "talloc.h"
void kstr_new(kstr *k)
{
*k = talloc(sizeof(**k));
die_if(!*k, out_of_memory);
(*k)->start = 0;
(*k)->length = 0;
(*k)->allocated_length = 0;
}
void kstr_del(kstr k)
{
tfree(k->start);
tfree(k);
}
void kstr_growto(kstr k, int len)
{
if (len > k->allocated_length) {
int nal = ((len | 7) + 1) * 2; /* twice the size, rounded up to the nearest 16 */
char *nstart = talloc(nal);
die_if(!nstart, out_of_memory);
memset(nstart, 'Y', nal); /* force bugs out */
memcpy(nstart, k->start, k->length);
tfree(k->start);
k->start = nstart;
k->allocated_length = nal;
}
}
void kstr_growby(kstr k, int len)
{
kstr_growto(k, len + k->length);
}
void kstr_getline(kstr k, FILE *f)
{
int l = 80; /* line length */
k->length = 0;
for (;;) {
char *rv;
kstr_growby(k, l);
clearerr(f);
rv = fgets(k->start + k->length, l, f);
if (!rv) {
return; /* end of file or error; check ferror() if you care */
}
k->length += strlen(k->start + k->length);
if (k->start[k->length - 1] == '\n') {
/* end of line */
k->start[k->length] = 'X'; /* force bugs out; isn't really part of string */
return;
}
l *= 2;
}
}
int kstr_read(kstr k, int fd, int maxlen)
{
int rv;
kstr_growto(k, maxlen);
rv = read(fd, k->start, maxlen);
if (rv <= 0) { /* error or eof */
k->length = 0;
return rv;
} else {
k->length = rv;
return rv;
}
}
void kstr_append(kstr k, char *s, int len)
{
kstr_growby(k, len);
memcpy(k->start + k->length, s, len);
k->length += len;
}
kstr.h/ 946249230 1069 1106 100644 712 `
/* kragen's strings */
/* patterned on djb's strallocs, but with less cutting-and-pasting and
* different error handling. */
typedef struct
{
char *start;
int length;
int allocated_length;
} *kstr;
/* to allocate and delete kstrs */
void kstr_new(kstr *k);
void kstr_del(kstr k);
/* These make sure the kstr has room for at least len bytes or at
* least len more bytes than it already has. */
void kstr_growto(kstr k, int len);
void kstr_growby(kstr k, int len);
/* Get a line from a stdio file. */
void kstr_getline(kstr k, FILE *f);
/* read bytes from a file descriptor. */
int kstr_read(kstr k, int fd, int maxlen);
/* append n bytes to a kstr. */
void kstr_append(kstr k, char *s, int len);
kstr_test.c/ 946228017 1069 1106 100644 322 `
#include <stdio.h>
#include "kstr.h"
#include "die.h"
char *input_error = "input error at %f:%l: %e";
int main()
{
kstr s;
kstr_new(&s);
while (!feof(stdin)) {
kstr_getline(s, stdin);
die_if(ferror(stdin), input_error);
fwrite(s->start, s->length, 1, stdout);
}
kstr_del(s);
return 0;
}
talloc.c/ 946245223 1069 1106 100644 384 `
#include <stdlib.h>
#include <stdio.h>
#include "talloc.h" /* for mismatch warnings */
/* to turn on tracing:
#define tracing /* */
void *talloc(int n)
{
void *rv = malloc(n);
#ifdef tracing
fprintf(stderr, "0x%08x: %d bytes\n", (unsigned)rv, n);
#endif
return rv;
}
void tfree(void *p)
{
#ifdef tracing
fprintf(stderr, "0x%08x: freed\n", (unsigned)p);
#endif
free(p);
}
talloc.h/ 946240307 1069 1106 100644 74 `
/* tracing malloc and free */
void *talloc(int n);
void tfree(void *p);
mem-used/ 946241116 1069 1106 100644 510 `
#!/var/u/sittler/bin/perl -w
use strict;
# analyze memory usage trace from talloc.
my %blocks;
my $total = 0;
while (<>) {
printf "%9d %s", $total, $_;
if (/^(0x[0-9a-f]+): (\d+) bytes$/) {
if (exists $blocks{$1}) {
warn "Uh-oh: $1 allocated twice without intervening free\n";
} else {
$blocks{$1} = $2;
$total += $2;
}
} elsif (/^(0x[0-9a-f]+): freed$/) {
next if $1 eq '0x00000000';
$total -= $blocks{$1};
delete $blocks{$1};
}
}
print "Final: $total\n";
chat-server.c/ 946252058 1069 1106 100644 7679 `
#include <sys/types.h>
#include <sys/socket.h>
#include <errno.h>
#include <stdio.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <unistd.h>
#include <fcntl.h>
#include <signal.h>
#include "event.h"
#include "kstr.h"
#include "die.h"
eventset e;
kstr client_list;
typedef struct client_info
{
int connected;
struct sockaddr_in sin;
kstr outbuf;
int outbufp;
} client_info;
client_info *get_cip(int fd)
{
return ((client_info*)client_list->start) + fd;
}
void handle_disconnection(int fd)
{
client_info *cip = get_cip(fd);
cip->connected = 0;
kstr_del(cip->outbuf);
on_event_nop(&e.read, fd);
on_event_nop(&e.write, fd);
close(fd);
}
void write_queued_data(int fd)
{
client_info *cip = get_cip(fd);
int rv;
die_if(!cip->connected, "Damn event handler called on disconnected client");
die_if(cip->outbufp > cip->outbuf->length, "outbufp out of range (%c)");
rv = write(fd,
cip->outbuf->start + cip->outbufp,
cip->outbuf->length - cip->outbufp);
if (rv < 0) {
fprintf(stderr, "error writing to client %d (%s): ", fd,
inet_ntoa(cip->sin.sin_addr));
perror("closing connection");
handle_disconnection(fd);
} else {
cip->outbufp += rv;
if (cip->outbufp == cip->outbuf->length) {
cip->outbufp = 0;
cip->outbuf->length = 0;
on_event_nop(&e.write, fd);
} else {
if (cip->outbufp > 15*cip->outbuf->length/16) {
/* time to move it back to the beginning of the buffer */
memcpy(cip->outbuf->start, cip->outbuf->start+cip->outbufp,
cip->outbuf->length - cip->outbufp);
cip->outbuf->length -= cip->outbufp;
cip->outbufp = 0;
}
}
}
}
char lostmsg[] = "(Lost messages)\r\n";
int queuelimit = 50 * 1024; /* no more than 50K of data for a client at a time,
possibly plus the length of lostmsg */
/* The reason for the draconian policy that follows: if you have three
clients connected to a chat server (or router, or switch, or what
have you), and each client decides when it wants to read data from
the server and when it wants to send data to the server, there are
some nasty possibilities. If one of the clients decides to stop
reading (for example, because it's been unplugged, suspended, lost
its network connection), while the other clients are still sending
stuff to the server, you have to decide what to do. Do you:
a) buffer up the information waiting for the slow client
indefinitely? This has the problem that it can require arbitrarily
large amounts of memory on the server, and since no physical server
has an infinite memory, there has to be a backup plan!
b) stop reading from other clients until the tardy client reads its
messages? This means that the other two clients can't communicate
with one another through the server until the tardy client wakes
up. If this sort of thing is the expected result of a client
computer crashing, we have a problem!
c) throw away messages that end up in the tardy client's outgoing
queue beyond a certain limit? That's the approach I adopted here,
because while it's evil, the other two policies are worse.
Other networks can take other approaches; two are notable.
The phone network avoids this problem by not allowing the clients
to decide when they want to read data from the switch; there's a
constant data rate of 64kbps. This has some problems when it comes
to many-to-many communications, but it works tolerably for
one-to-one.
The other notable approach is hop-by-hop acknowledgment; when a
client A wants to communicate with another client B, it sends some
data for B to the server. Once A sends it, it refrains from
sending any more data for B until the server tells A that the data
has been successfully delivered to B. So the amount of data that
may have to be buffered is constrained by the number of clients:
two bufferfuls for each communicating pair.
Neither of these is applicable to the present problem.
Once this decision has been reached, the only question is how much
data to buffer for each client. I'd like a reasonably cheap 512M
machine to be able to serve 10,000 simultaneous clients under the
worst circumstances. That means each client can require up to 50K
of data buffered. [kstr's allocation policies, select()'s inherent
slowness, and the limitations of fd_sets probably prohibit this
anyway, by the way.]
*/
void queue_data(int fd, char *s, int len)
{
client_info *cip = get_cip(fd);
die_if(!cip->connected, "Attempt to send to disconnected client");
if (cip->outbuf->length + len > queuelimit) {
if (cip->outbuf->length < queuelimit) {
kstr_append(cip->outbuf, lostmsg, sizeof(lostmsg)-1);
} else {
/* silently drop it */
}
} else {
kstr_append(cip->outbuf, s, len);
}
on_event(&e.write, fd, write_queued_data);
}
void queue_string(int fd, char *s)
{
queue_data(fd, s, strlen(s));
}
kstr rbuf;
void handle_client_data(int fd)
{
int rv;
rv = kstr_read(rbuf, fd, 8192);
if (rv < 0) {
fprintf(stderr, "client fd %d:", fd);
perror("read error");
} else if (rv == 0) {
handle_disconnection(fd);
} else {
int i;
client_info *cip = get_cip(0);
for (i = 0; i != e.read.size; i++) {
if (cip[i].connected) {
queue_string(i, "From ");
queue_string(i, inet_ntoa(cip[fd].sin.sin_addr));
queue_string(i, ": ");
queue_data(i, rbuf->start, rbuf->length);
}
}
}
}
void new_client_conn(int listenfd)
{
struct sockaddr_in addr;
socklen_t socklen = sizeof(addr);
client_info *cip;
int space_to_allocate;
int nc = accept(listenfd, (struct sockaddr*)&addr, &socklen);
/* set non-blocking mode, as with FIONBIO or O_NONBLOCK */
/* at least on Solaris 7, writing to TCP sockets not marked as
* nonblocking that select as writable can hang you! It happened to
* me writing 8224 bytes to a rather drenched socket; control-C
* caused write() to return with 8192. */
fcntl(nc, F_SETFL, fcntl(nc, F_GETFL, 0) | O_NDELAY);
kstr_growto(client_list, (nc+1) * sizeof(struct client_info));
space_to_allocate = (nc+1) * sizeof(struct client_info) - client_list->length;
memset(client_list->start + client_list->length, '\0', space_to_allocate);
client_list->length += space_to_allocate;
cip = ((client_info*)client_list->start) + nc;
cip->connected = 1;
cip->sin = addr;
kstr_new(&cip->outbuf);
cip->outbufp = 0;
on_event(&e.read, nc, handle_client_data);
queue_string(nc, "Hello there ");
queue_string(nc, inet_ntoa(addr.sin_addr));
queue_string(nc, "\n");
}
int open_server_socket()
{
int fd = socket(PF_INET, SOCK_STREAM, 0);
int rv;
int one = 1;
struct sockaddr_in addr;
setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &one, sizeof one);
memset((char*)&addr, '\0', sizeof(addr));
addr.sin_family = AF_INET;
addr.sin_port = htons(17224); /* 'CH' */
addr.sin_addr.s_addr = INADDR_ANY;
rv = bind(fd, (struct sockaddr*)&addr, sizeof(addr));
die_if(rv<0, "bind failed: %e");
rv = listen(fd, 5);
die_if(rv<0, "listen failed: %e");
return fd;
}
void end_server(int fd)
{
kstr_del(client_list);
kstr_del(rbuf);
exit(0);
}
int main()
{
int s = open_server_socket();
kstr_new(&client_list);
kstr_new(&rbuf);
/* normally these signals, from writes on closed sockets, kill the server */
sigignore(SIGPIPE);
init_eventset(&e);
on_event(&e.read, s, new_client_conn);
on_event(&e.read, 0, end_server);
for (;;) {
handle_events(&e);
}
die_if(1 + 1 == 2, "Can't happen at %f:%l");
return 0;
}
* Previous message: APLish basic stats program in Perl
* Next message: chat-server patch
* Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]