SocketServer.py 16.6 KB
Newer Older
Guido van Rossum's avatar
Guido van Rossum committed
1 2 3 4
"""Generic socket server classes.

This module tries to capture the various aspects of defining a server:

5 6
For socket-based servers:

Guido van Rossum's avatar
Guido van Rossum committed
7
- address family:
8 9 10
        - AF_INET: IP (Internet Protocol) sockets (default)
        - AF_UNIX: Unix domain sockets
        - others, e.g. AF_DECNET are conceivable (see <socket.h>
Guido van Rossum's avatar
Guido van Rossum committed
11
- socket type:
12 13
        - SOCK_STREAM (reliable stream, e.g. TCP)
        - SOCK_DGRAM (datagrams, e.g. UDP)
14 15 16

For request-based servers (including socket-based):

Guido van Rossum's avatar
Guido van Rossum committed
17
- client address verification before further looking at the request
18 19
        (This is actually a hook for any processing that needs to look
         at the request before anything else, e.g. logging)
Guido van Rossum's avatar
Guido van Rossum committed
20
- how to handle multiple requests:
21 22 23
        - synchronous (one request is handled at a time)
        - forking (each request is handled by a new process)
        - threading (each request is handled by a new thread)
Guido van Rossum's avatar
Guido van Rossum committed
24 25 26 27 28 29

The classes in this module favor the server type that is simplest to
write: a synchronous TCP/IP server.  This is bad class design, but
save some typing.  (There's also the issue that a deep class hierarchy
slows down method lookups.)

30
There are five classes in an inheritance diagram, four of which represent
Guido van Rossum's avatar
Guido van Rossum committed
31 32
synchronous servers of four types:

33 34 35 36 37
        +------------+
        | BaseServer |
        +------------+
              |
              v
38 39 40 41 42 43 44 45
        +-----------+        +------------------+
        | TCPServer |------->| UnixStreamServer |
        +-----------+        +------------------+
              |
              v
        +-----------+        +--------------------+
        | UDPServer |------->| UnixDatagramServer |
        +-----------+        +--------------------+
Guido van Rossum's avatar
Guido van Rossum committed
46

47
Note that UnixDatagramServer derives from UDPServer, not from
Guido van Rossum's avatar
Guido van Rossum committed
48 49
UnixStreamServer -- the only difference between an IP and a Unix
stream server is the address family, which is simply repeated in both
50
unix server classes.
Guido van Rossum's avatar
Guido van Rossum committed
51 52 53 54 55

Forking and threading versions of each type of server can be created
using the ForkingServer and ThreadingServer mix-in classes.  For
instance, a threading UDP server class is created as follows:

56
        class ThreadingUDPServer(ThreadingMixIn, UDPServer): pass
Guido van Rossum's avatar
Guido van Rossum committed
57

58 59
The Mix-in class must come first, since it overrides a method defined
in UDPServer!
Guido van Rossum's avatar
Guido van Rossum committed
60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89

To implement a service, you must derive a class from
BaseRequestHandler and redefine its handle() method.  You can then run
various versions of the service by combining one of the server classes
with your request handler class.

The request handler class must be different for datagram or stream
services.  This can be hidden by using the mix-in request handler
classes StreamRequestHandler or DatagramRequestHandler.

Of course, you still have to use your head!

For instance, it makes no sense to use a forking server if the service
contains state in memory that can be modified by requests (since the
modifications in the child process would never reach the initial state
kept in the parent process and passed to each child).  In this case,
you can use a threading server, but you will probably have to use
locks to avoid two requests that come in nearly simultaneous to apply
conflicting changes to the server state.

On the other hand, if you are building e.g. an HTTP server, where all
data is stored externally (e.g. in the file system), a synchronous
class will essentially render the service "deaf" while one request is
being handled -- which may be for a very long time if a client is slow
to reqd all the data it has requested.  Here a threading or forking
server is appropriate.

In some cases, it may be appropriate to process part of a request
synchronously, but to finish processing in a forked child depending on
the request data.  This can be implemented by using a synchronous
90
server and doing an explicit fork in the request handler class
Guido van Rossum's avatar
Guido van Rossum committed
91 92 93 94 95 96 97 98 99
handle() method.

Another approach to handling multiple simultaneous requests in an
environment that supports neither threads nor fork (or where these are
too expensive or inappropriate for the service) is to maintain an
explicit table of partially finished requests and to use select() to
decide which request to work on next (or whether to handle a new
incoming request).  This is particularly important for stream services
where each client can potentially be connected for a long time (if
100
threads or subprocesses cannot be used).
Guido van Rossum's avatar
Guido van Rossum committed
101 102 103 104 105 106 107 108 109 110

Future work:
- Standard classes for Sun RPC (which uses either UDP or TCP)
- Standard mix-in classes to implement various authentication
  and encryption schemes
- Standard framework for select-based multiplexing

XXX Open problems:
- What to do with out-of-band data?

111 112 113 114 115 116 117 118
BaseServer:
- split generic "request" functionality out into BaseServer class.
  Copyright (C) 2000  Luke Kenneth Casson Leighton <lkcl@samba.org>

  example: read entries from a SQL database (requires overriding
  get_request() to return a table entry from the database).
  entry is processed by a RequestHandlerClass.

Guido van Rossum's avatar
Guido van Rossum committed
119 120
"""

121
# Author of the BaseServer patch: Luke Kenneth Casson Leighton
Guido van Rossum's avatar
Guido van Rossum committed
122

123
__version__ = "0.3"
Guido van Rossum's avatar
Guido van Rossum committed
124 125 126 127 128 129


import socket
import sys
import os

130 131 132 133 134 135 136
__all__ = ["TCPServer","UDPServer","ForkingUDPServer","ForkingTCPServer",
           "ThreadingUDPServer","ThreadingTCPServer","BaseRequestHandler",
           "StreamRequestHandler","DatagramRequestHandler"]
if hasattr(socket, "AF_UNIX"):
    __all__.extend(["UnixStreamServer","UnixDatagramServer",
                    "ThreadingUnixStreamServer",
                    "ThreadingUnixDatagramServer"])
Guido van Rossum's avatar
Guido van Rossum committed
137

138
class BaseServer:
Guido van Rossum's avatar
Guido van Rossum committed
139

140
    """Base class for server classes.
Guido van Rossum's avatar
Guido van Rossum committed
141 142 143 144 145

    Methods for the caller:

    - __init__(server_address, RequestHandlerClass)
    - serve_forever()
146
    - handle_request()  # if you do not use serve_forever()
147
    - fileno() -> int   # for select()
Guido van Rossum's avatar
Guido van Rossum committed
148 149 150 151 152 153 154

    Methods that may be overridden:

    - server_bind()
    - server_activate()
    - get_request() -> request, client_address
    - verify_request(request, client_address)
155
    - server_close()
Guido van Rossum's avatar
Guido van Rossum committed
156 157 158 159 160 161 162 163 164 165 166 167
    - process_request(request, client_address)
    - handle_error()

    Methods for derived classes:

    - finish_request(request, client_address)

    Class variables that may be overridden by derived classes or
    instances:

    - address_family
    - socket_type
168
    - reuse_address
Guido van Rossum's avatar
Guido van Rossum committed
169 170 171 172 173 174 175 176 177

    Instance variables:

    - RequestHandlerClass
    - socket

    """

    def __init__(self, server_address, RequestHandlerClass):
178 179 180
        """Constructor.  May be extended, do not override."""
        self.server_address = server_address
        self.RequestHandlerClass = RequestHandlerClass
Guido van Rossum's avatar
Guido van Rossum committed
181 182

    def server_activate(self):
183
        """Called by constructor to activate the server.
Guido van Rossum's avatar
Guido van Rossum committed
184

185
        May be overridden.
Guido van Rossum's avatar
Guido van Rossum committed
186

187
        """
188
        pass
Guido van Rossum's avatar
Guido van Rossum committed
189 190

    def serve_forever(self):
191 192 193
        """Handle one request at a time until doomsday."""
        while 1:
            self.handle_request()
Guido van Rossum's avatar
Guido van Rossum committed
194 195 196 197 198 199 200 201 202 203 204 205 206

    # The distinction between handling, getting, processing and
    # finishing a request is fairly arbitrary.  Remember:
    #
    # - handle_request() is the top-level call.  It calls
    #   get_request(), verify_request() and process_request()
    # - get_request() is different for stream or datagram sockets
    # - process_request() is the place that may fork a new process
    #   or create a new thread to finish the request
    # - finish_request() instantiates the request handler class;
    #   this constructor will handle the request all by itself

    def handle_request(self):
207
        """Handle one request, possibly blocking."""
208 209 210 211
        try:
            request, client_address = self.get_request()
        except socket.error:
            return
212 213 214 215 216
        if self.verify_request(request, client_address):
            try:
                self.process_request(request, client_address)
            except:
                self.handle_error(request, client_address)
Guido van Rossum's avatar
Guido van Rossum committed
217 218

    def verify_request(self, request, client_address):
219
        """Verify the request.  May be overridden.
Guido van Rossum's avatar
Guido van Rossum committed
220

221
        Return true if we should proceed with this request.
Guido van Rossum's avatar
Guido van Rossum committed
222

223 224
        """
        return 1
Guido van Rossum's avatar
Guido van Rossum committed
225 226

    def process_request(self, request, client_address):
227
        """Call finish_request.
Guido van Rossum's avatar
Guido van Rossum committed
228

229
        Overridden by ForkingMixIn and ThreadingMixIn.
Guido van Rossum's avatar
Guido van Rossum committed
230

231 232
        """
        self.finish_request(request, client_address)
Guido van Rossum's avatar
Guido van Rossum committed
233

234 235 236 237 238 239 240 241
    def server_close(self):
        """Called to clean-up the server.

        May be overridden.

        """
        pass

Guido van Rossum's avatar
Guido van Rossum committed
242
    def finish_request(self, request, client_address):
243 244
        """Finish one request by instantiating RequestHandlerClass."""
        self.RequestHandlerClass(request, client_address, self)
Guido van Rossum's avatar
Guido van Rossum committed
245 246

    def handle_error(self, request, client_address):
247
        """Handle an error gracefully.  May be overridden.
Guido van Rossum's avatar
Guido van Rossum committed
248

249
        The default is to print a traceback and continue.
Guido van Rossum's avatar
Guido van Rossum committed
250

251 252 253 254 255
        """
        print '-'*40
        print 'Exception happened during processing of request from',
        print client_address
        import traceback
256
        traceback.print_exc() # XXX But this goes to stderr!
257
        print '-'*40
Guido van Rossum's avatar
Guido van Rossum committed
258 259


260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360
class TCPServer(BaseServer):

    """Base class for various socket-based server classes.

    Defaults to synchronous IP stream (i.e., TCP).

    Methods for the caller:

    - __init__(server_address, RequestHandlerClass)
    - serve_forever()
    - handle_request()  # if you don't use serve_forever()
    - fileno() -> int   # for select()

    Methods that may be overridden:

    - server_bind()
    - server_activate()
    - get_request() -> request, client_address
    - verify_request(request, client_address)
    - process_request(request, client_address)
    - handle_error()

    Methods for derived classes:

    - finish_request(request, client_address)

    Class variables that may be overridden by derived classes or
    instances:

    - address_family
    - socket_type
    - request_queue_size (only for stream sockets)
    - reuse_address

    Instance variables:

    - server_address
    - RequestHandlerClass
    - socket

    """

    address_family = socket.AF_INET

    socket_type = socket.SOCK_STREAM

    request_queue_size = 5

    allow_reuse_address = 0

    def __init__(self, server_address, RequestHandlerClass):
        """Constructor.  May be extended, do not override."""
        BaseServer.__init__(self, server_address, RequestHandlerClass)
        self.socket = socket.socket(self.address_family,
                                    self.socket_type)
        self.server_bind()
        self.server_activate()

    def server_bind(self):
        """Called by constructor to bind the socket.

        May be overridden.

        """
        if self.allow_reuse_address:
            self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        self.socket.bind(self.server_address)

    def server_activate(self):
        """Called by constructor to activate the server.

        May be overridden.

        """
        self.socket.listen(self.request_queue_size)

    def server_close(self):
        """Called to clean-up the server.

        May be overridden.

        """
        self.socket.close()

    def fileno(self):
        """Return socket file number.

        Interface required by select().

        """
        return self.socket.fileno()

    def get_request(self):
        """Get the request and client address from the socket.

        May be overridden.

        """
        return self.socket.accept()


Guido van Rossum's avatar
Guido van Rossum committed
361 362 363 364
class UDPServer(TCPServer):

    """UDP server class."""

365 366
    allow_reuse_address = 0

Guido van Rossum's avatar
Guido van Rossum committed
367 368 369 370 371
    socket_type = socket.SOCK_DGRAM

    max_packet_size = 8192

    def get_request(self):
372 373 374 375 376 377
        data, client_addr = self.socket.recvfrom(self.max_packet_size)
        return (data, self.socket), client_addr

    def server_activate(self):
        # No need to call listen() for UDP.
        pass
Guido van Rossum's avatar
Guido van Rossum committed
378 379 380 381 382 383 384


class ForkingMixIn:

    """Mix-in class to handle each request in a new process."""

    active_children = None
385
    max_children = 40
Guido van Rossum's avatar
Guido van Rossum committed
386 387

    def collect_children(self):
388 389
        """Internal routine to wait for died children."""
        while self.active_children:
390 391 392 393 394 395
            if len(self.active_children) < self.max_children:
                options = os.WNOHANG
            else:
                # If the maximum number of children are already
                # running, block while waiting for a child to exit
                options = 0
396
            try:
397
                pid, status = os.waitpid(0, options)
398 399
            except os.error:
                pid = None
400 401
            if not pid: break
            self.active_children.remove(pid)
Guido van Rossum's avatar
Guido van Rossum committed
402 403

    def process_request(self, request, client_address):
404 405 406 407 408 409 410 411 412 413 414 415 416
        """Fork a new subprocess to process the request."""
        self.collect_children()
        pid = os.fork()
        if pid:
            # Parent process
            if self.active_children is None:
                self.active_children = []
            self.active_children.append(pid)
            return
        else:
            # Child process.
            # This must never return, hence os._exit()!
            try:
417
                self.server_close()
418 419 420 421 422 423 424 425
                self.finish_request(request, client_address)
                os._exit(0)
            except:
                try:
                    self.handle_error(request,
                                      client_address)
                finally:
                    os._exit(1)
Guido van Rossum's avatar
Guido van Rossum committed
426 427 428 429 430 431


class ThreadingMixIn:
    """Mix-in class to handle each request in a new thread."""

    def process_request(self, request, client_address):
432
        """Start a new thread to process the request."""
433 434 435 436
        import threading
        t = threading.Thread(target = self.finish_request,
                             args = (request, client_address))
        t.start()
Guido van Rossum's avatar
Guido van Rossum committed
437 438 439 440 441 442 443 444


class ForkingUDPServer(ForkingMixIn, UDPServer): pass
class ForkingTCPServer(ForkingMixIn, TCPServer): pass

class ThreadingUDPServer(ThreadingMixIn, UDPServer): pass
class ThreadingTCPServer(ThreadingMixIn, TCPServer): pass

445 446 447 448 449 450 451 452 453 454 455
if hasattr(socket, 'AF_UNIX'):

    class UnixStreamServer(TCPServer):
        address_family = socket.AF_UNIX

    class UnixDatagramServer(UDPServer):
        address_family = socket.AF_UNIX

    class ThreadingUnixStreamServer(ThreadingMixIn, UnixStreamServer): pass

    class ThreadingUnixDatagramServer(ThreadingMixIn, UnixDatagramServer): pass
Guido van Rossum's avatar
Guido van Rossum committed
456 457 458 459 460 461 462 463 464 465 466 467

class BaseRequestHandler:

    """Base class for request handler classes.

    This class is instantiated for each request to be handled.  The
    constructor sets the instance variables request, client_address
    and server, and then calls the handle() method.  To implement a
    specific service, all you need to do is to derive a class which
    defines a handle() method.

    The handle() method can find the request as self.request, the
468
    client address as self.client_address, and the server (in case it
Guido van Rossum's avatar
Guido van Rossum committed
469 470 471 472 473 474 475
    needs access to per-server information) as self.server.  Since a
    separate instance is created for each request, the handle() method
    can define arbitrary other instance variariables.

    """

    def __init__(self, request, client_address, server):
476 477 478 479 480 481 482 483 484
        self.request = request
        self.client_address = client_address
        self.server = server
        try:
            self.setup()
            self.handle()
            self.finish()
        finally:
            sys.exc_traceback = None    # Help garbage collection
Guido van Rossum's avatar
Guido van Rossum committed
485 486

    def setup(self):
487
        pass
Guido van Rossum's avatar
Guido van Rossum committed
488 489

    def __del__(self):
490
        pass
Guido van Rossum's avatar
Guido van Rossum committed
491 492

    def handle(self):
493
        pass
Guido van Rossum's avatar
Guido van Rossum committed
494 495

    def finish(self):
496
        pass
Guido van Rossum's avatar
Guido van Rossum committed
497 498 499 500 501 502 503 504 505 506 507 508 509 510


# The following two classes make it possible to use the same service
# class for stream or datagram servers.
# Each class sets up these instance variables:
# - rfile: a file object from which receives the request is read
# - wfile: a file object to which the reply is written
# When the handle() method returns, wfile is flushed properly


class StreamRequestHandler(BaseRequestHandler):

    """Define self.rfile and self.wfile for stream sockets."""

511 512 513 514 515 516 517 518 519 520
    # Default buffer sizes for rfile, wfile.
    # We default rfile to buffered because otherwise it could be
    # really slow for large data (a getc() call per byte); we make
    # wfile unbuffered because (a) often after a write() we want to
    # read and we need to flush the line; (b) big writes to unbuffered
    # files are typically optimized by stdio even when big reads
    # aren't.
    rbufsize = -1
    wbufsize = 0

Guido van Rossum's avatar
Guido van Rossum committed
521
    def setup(self):
522
        self.connection = self.request
523 524
        self.rfile = self.connection.makefile('rb', self.rbufsize)
        self.wfile = self.connection.makefile('wb', self.wbufsize)
Guido van Rossum's avatar
Guido van Rossum committed
525 526

    def finish(self):
527
        self.wfile.flush()
528 529
        self.wfile.close()
        self.rfile.close()
Guido van Rossum's avatar
Guido van Rossum committed
530 531 532 533 534 535 536


class DatagramRequestHandler(BaseRequestHandler):

    """Define self.rfile and self.wfile for datagram sockets."""

    def setup(self):
537 538 539 540
        import StringIO
        self.packet, self.socket = self.request
        self.rfile = StringIO.StringIO(self.packet)
        self.wfile = StringIO.StringIO(self.packet)
Guido van Rossum's avatar
Guido van Rossum committed
541 542

    def finish(self):
543
        self.socket.sendto(self.wfile.getvalue(), self.client_address)