Scale to any load

Scale to any load

SurgeNews supports several methods for spreading the load over multiple servers. The two basic methods are:

1) Split the groups. The news is split by group name onto back end servers, and you use proxy servers to access them.

2) Split by mesasge ID. The news by message id and use front end index servers to proxy them. (experimental)

Split the groups (recommended)

In proxy mode SurgeNews acts as a router connecting the user almost directly to the upstream or back end server once they specify which news group they want to read, in this way you can have a single point of connection for incoming users, but actually have the news stored on any number of back end news servers.

Typically you would use this to spread the storage & processor load over multiple cheap servers, typically one for mp3's one for 'dvd's one for 'other binaries' and one for 'non binaries'. As the servers are not taking a full feed you can use cheap IDE drives and medium speed processors.

The widearea quota limits in SurgeNews let you apply user quota limits over these multiple servers.

Config setting: global.htm#g_proxy

Feed server:
g_feed host="mp3.backend.com" groups="*mp3*" type="full"
g_feed host="dvd.backend.com" groups="*dvd*,!*mp3*" type="full"
g_feed host="text.backend.com" groups="*,!*dvd*,!*mp3*" type="full"
Proxy server (s):
g_proxy host="mp3.backend.com" groups="*mp3*" 
g_proxy host="dvd.backend.com" groups="*dvd*,!*mp3*" 
g_proxy host="text.backend.com" groups="*,!*dvd*,!*mp3*" 
Split by message ID (Experimental)

The incoming messages are split randomly between 'n' back end servers. Front end servers then store only the item headers and fetch the bodies off the back end as needed (caching in a circular disk cache to further offload the back end systems)

SurgeNews supports this mechanism in several layouts allowing you to make a system of exactly the size you need, the simplist form involves as few as two servers. In this mode one server splits the feed and stores 'half' of it while forwarding the other half to the secondary system, which stores that half, and sends a header feed back to the master. Then both servers appear to have all items, but really each server only needed to store 'half' the full incoming load.

This can be extended across any number of servers.

You can also split the 'feeding' server off entirely and run it stand alone, or even use your existing software if it knows how to devide a feed up.

Lastly you can run any number of 'front' end servers seperately instead of both storing part of the feed, and providing the front end server functions on the same sysetms.

So for a typical ISP you might simply run 2-3 servers sharing the entire load, and for a specialist news hosting farm you might run 1 feed servers, 4-n back end servers, and 4-m front end servers.

In addition back end servers can be 'replicated' and front end servers will automatically 'fail over' and 'load share' to the duplicated back end servers.

Features:

Servers can be the feed master, and or a back end and or a front end all at the same time.

User quota's and limits are calcuated using a quota host so usage is measured across all systems.

Bodies are 'cached' to off load back ends.

Full xover and header data exists on each front end to reduce the back end load.

Expiration is automatic based on the item ranges of the back end servers.

The Pull module can fetch existing items when adding a new 'front end' server to a cluster.

Failover to multiple back end servers from front end servers.

Configuration examples, only relevant settings are shown:

Sharing the load between two servers

Master server (master.xyz.com)
g_feed name="Hashed" host="slave.xyz.com" groups="*" type="full" hash="1" 
g_feed name="Headers" host="slave.xyz.com" groups="*" type="header" hash="0" expire="true"
g_feed_hash "2" 
g_feed_hash_me "0" 
g_pile groups="*" use="20%" bodycache="TRUE" 
g_pull host="slave.xyz.com" groups="*" bodyonly="TRUE" expire="true"       
g_replicate "slave.xyz.com"
vdomain name="master.xyz.com" 
Slave server (slave.xyz.com)
g_feed name="Hashed" host="master.xyz.com" groups="*" type="header" 
g_pile groups="*" use="20%" bodycache="TRUE" 
g_pull host="master.xyz.com" groups="*" bodyonly="TRUE" expire="true"     
g_replicate "master.xyz.com" 
g_replicate_post "master.xyz.com"
vdomain name="slave.xyz.com" 
Also ensure all servers can 'read' news off each other without authenticating.

In the case of 3 or more servers you must specify all the other servers in the g_replicate setting, and you must add an additional g_feed and g_pull entry for each other system.

Using N front ends, and M back ends, and a feed server.

Feed Server
g_feed name="Hashed" host="backend1.xyz.com" groups="*" type="full" hash="0" 
g_feed name="Hashed" host="backend2.xyz.com" groups="*" type="full" hash="1" 
g_feed name="Hashed" host="backend3.xyz.com" groups="*" type="full" hash="2" 
g_feed_hash "3" 
g_feed_hash_me "99" 
g_pile groups="*" use="20%" bodycache="TRUE" 
vdomain name="master.xyz.com" 
Back End Serers
g_feed name="Headers" host="front1.xyz.com" groups="*" type="header" 
g_feed name="Headers" host="front2.xyz.com" groups="*" type="header" 
g_feed name="Headers" host="front3.xyz.com" groups="*" type="header" 
g_replicate "master.xyz.com"
vdomain name="backend1.xyz.com" 
Front End Servers
g_feed host="master.xyz.com" groups="*" type="localonly"
g_pile groups="*" use="20%" bodycache="TRUE" 
g_pull host="backend1.xyz.com" groups="*" bodyonly="TRUE" expire="true"
g_pull host="backend2.xyz.com" groups="*" bodyonly="TRUE" expire="true"
g_pull host="backend3.xyz.com" groups="*" bodyonly="TRUE" expire="true"
g_replicate "backend1.xyz.com,backend2.xyz.com,backend3.xyz.com" 
g_replicate_post "master.xyz.com"
vdomain name="front1.xyz.com" 