Shallow Rampa - Atlantis Clouds

Atlantis Clouds

by dixie on Date unknown

Tagged as: data, chat.

About

For some period of time (about 2–3 months) there was running who snapshots which have been used as input for clouds pictures here. Images are produced by wordcloud program included at diagrams haskell package.

Size of letters determines higher sum of measured value over whole snapshoting interval.

Nicknames cloud

As measured value is how often given person was online. SVG With Better Resolution 5MB

Nicks

Rooms cloud

As measured value is how many people are at room.

Rooms

Levels cloud

As measured value is how much people of given level were online.

Levels

Shallow Rampa - Atlantis Social Network

Atlantis Social Network

by dixie on Date unknown

Tagged as: data, chat.

About

For some period of time (about 2–3 months) there was running who snapshots which have been used as input for social network here. Graphical output of graph is procuded by graphviz and pure graph data are constructed by haskell program.

Each Node represents a nickname. Edge are constructed based on frequency of the nickname pairs at the same room and time (with some variantions below)

Edge like this (origin Nick1 and destination Nick2):

[Nick1]--->[Nick2]

Above relation means that the most times is nick1 with nick2 at the same room.

For viewing SVG capable image viewer is needed, or Firefox.

Social Network Groups

Here for each person is taken only most cooroom person.

• Atlantis Social Network in SVG - 500KB

Social Network Pairs

Here for each person is taken only most cooroom persons but for analysis only two people at time+room are considered (pairs).

• Atlantis Social Network Pairs in SVG - 500KB

Shallow Rampa - Dawb - DArcs Web Browser

Dawb - DArcs Web Browser

by dixie on Date unknown

Tagged as: darcs, app.

Overview

Dawb is simple web based browser for the Darcs repositories. It allows quick check of repositories which are hosted on pure HTTP server without DarcsWeb or anything similar (patchtag, darcsden). Sometimes I need to check how alive is the some project without darcs command near the hand.

Accessible on:

http://handra.rampa.sk/dawb/changes

It serves also RSS feed.

Some operations are slow because:

• the dawb code is not optimized
• some darcs operations are slow on big repositories (e.g. darcs log <element>)
• served on shared low resource virtual machine

Complex queries can be slow and end fish timeout, usually the darcs log some-element

Features

• View repository changes (darcs log)
• Regex and element filter for changes view
• View files with syntax highlight (highlighting-kate)
• View changes with syntax highlight
• All information stored in URL - can be sent over IM or EMAIL
• RSS feed for repository changes

Known issues

View file functionality doesn’t work on repositories hosted on patchtag or darcsden. The reason is in assumption of dawb how to access file over HTTP.

Examples

Example repositories:

• http://ui3.info/darcs/epub-metadata/
• http://darcs.net/screened/
• http://darcs.net/reviewed/
• http://darcs.haskell.org/ghc
• http://code.haskell.org/hoogle
• http://code.haskell.org/xmonad
• http://code.haskell.org/XMonadContrib
• http://code.haskell.org/gtk2hs/
• http://darcs.nomeata.de/darcswatch/
• http://code.haskell.org/lambdabot

RSS feeds:

• RSS for http://darcs.net/
• RSS for http://wiki.darcs.net/

Installation

Software is not prepared for general usage and it has all configuration hardcoded. It starts local HTTP service on http://localhost:3000/changes after start.

$ darcs get --lazy http://darcsden.com/dixie/dawb
$ cd dawb
$ cabal install
$ dawb

Please be aware that it is still ALPHA quality software.

Download

• Source code repository (darcs) at http://darcsden.com/dixie/dawb

Shallow Rampa - Jabber Telnet Bot Gateway

Jabber Telnet Bot Gateway

by dixie on Date unknown

Tagged as: chat, xmpp, app.

Overview

Jabber Bot provides interface for telnet connection over XMPP.

It allows interaction with MUDs or Talkers over the IM clients (pidgin, gajim, gtalk, etc).

Overview Picture

Developed for Atlantis Talker. Atlantis Talker uses Slovak/English languages mix. You can use it over the JID atlantis@atlantis.talker.sk on any XMPP network like gmail chat.

The development of Jabber Telnet Bot is discontinued.

Source Repository

• https://sourceforge.net/projects/jabbertelnetbot/

FAQ

• Q: Is this transport for XMPP server which allows interactions with other XMPP servers and user can add Atlantis/Talker user into Contact List ?
• A: No. This is bot which is something like a gateway over which user can create telnet session and interact with Atlantis or other telnet service.

Known Issues

• To have working bot agaist latest jabberd 2.4 it is needed to do some workarounds:
  • disable digest-md5 http://jabberd2.xiaoka.com/ticket/233
  • disable SASL in the bot

Download

• jabbertelnetbot–0.4.tgz
• jabbertelnetbot–0.3.tgz

Screenshots

Screenshot Pidgin (0.3)

Screenshot Gmail (.0.4)

Original Plan for next version 0.5

Actually with help of guys at #pidgin I found exact limit on pidgin for rich text messages:

        /* Awful hack to work around GtkIMHtml's inefficient rendering of messages with lots of formatting changes.
         * If a message has over 100 '<' characters, strip formatting before appending it. Hopefully nobody actually
         * needs that much formatting, anyway.
         */
        for (bracket = strchr(displaying, '<'); bracket && *(bracket + 1); bracket = strchr(bracket + 1, '<'))
                tag_count++;

        if (tag_count > 100) {
                char *tmp = displaying;
                displaying = purple_markup_strip_html(tmp);
                g_free(tmp);
        }

• I’ll add in next version support for this limit consideration during telnet->jabber message conversion.
• commands for control of telnet-jabber session, like “connect”, “disconnect” (allow to connect on any telnet server)

Installation (Source Release only)

Following dependencies are needed:

• Python 2.5.2
• Python xmpp 0.3.1

For debian-based distros (Ubuntu) type following for installing them:

$ sudo apt-get install python-xmpp python

Extract JabberTelnetBot-X.Y.tar.gz

$ tar xfvz JabberTelnetBot-X.Y.tar.gz

Configuration

It is necessary to setup credentials for Jabber Account which is used by robot for reading incomming commands.

Template configuration is at cfg/ directory and it is python source file.

# Atlantis host/port
ATL_HOST="atlantis.talker.sk"
ATL_PORT=23
# JID = login into jabber / PWD = plaintext password
JABBER_JID="meciar@gmail.com"
JABBER_PWD="nbusr123"
# This can be Set to None if JID contains all info
JABBER_SERVER=("talk.google.com", 5223)
# JABBER_SERVER=None

Usage/Start

For help:

$ ./jbtelbotd --help

For start:

$ ./jbtelbotd -d -v -c ./cfg/jbtelbot.conf

It starts Jabber Telnet Bot on foreground. You can stop it with CTRL+C on terminal.

It should connect and authorize to configured jabber server.

Usage/Chat

It is necessary add the bot into your other jabber account contact list.

When the bot appear as online it is possible send him messages which are send to established telnet connection. Any response from the telnet connection is sent back as message to given jabber user.

There is 1:1 mapping between jabber user and established telnet connection.

Shallow Rampa - LDIF - LDAP Data Interchange Format

LDIF - LDAP Data Interchange Format

by dixie on Date unknown

Tagged as: app, lib, tree.

Overview

ldif is Haskell library and tools for work with LDIF files. For the details about the LDIF please look LDIF on Wikipedia:

The LDAP Data Interchange Format (LDIF) is a standard plain text data
interchange format for representing LDAP (Lightweight Directory Access
Protocol) directory content and update requests. LDIF conveys
directory content as a set of records, one record for each object (or
entry). It represents update requests, such as Add, Modify, Delete,
and Rename, as a set of records, one record for each update request.

The parser from library is still not in 100% conformity with the LDIF related RFCs.

Installation

Use the HackageDB as usually.

$ cabal install HUnit
$ cabal install ldif

Please be aware that it is still ALPHA quality software.

Usage

Commands provided:

• ldif2html - LDIF file pretty printing to the HTML file
• diffLDIF - Calculates delta between two LDIF files
• ldifmodify - Replays delta
• ldifparse - Ignore, only for parser debugging purposes

diffLDIF

Calculates delta LDIF based on two given content LDIF files.

If the change LDIF is applied into LDAP directory with state of content src.ldif the state will be changed to dst.ldif.

Delta LDIF contains LDAP operations like add, delete, modify. It is similar to the patch of the unix diff but on the LDIF files.

Usage:

diffLDIF: Create delta LDIF between Source LDIF and Target LDIF

diffldif [FLAG]

  -? --help[=FORMAT]  Show usage information (optional format)
  -V --version        Show version information
  -v --verbose        Higher verbosity
  -q --quiet          Lower verbosity
  -s --srcFile=FILE   Source LDIF File
  -t --dstFile=FILE   Target LDIF File

In pictures, there is process of Changes LDIF construction (diff):

Process of applying changes LDIF into source LDIF (ldapmodify):

See example LDIF files below.

Fist input file - filesrc.ldif

dn: cn=The Postmaster,dc=example,dc=com
objectClass: organizationalRole
cn: The Postmaster
oldAttr: attrValue1
oldAttr: attrValue2

dn: cn=The Postmaster Remove,dc=example,dc=com
objectClass: organizationalRole
cn: The Postmaster Remove

Second input file - filedst.ldif

dn: cn=The Postmaster,dc=example,dc=com
objectClass: organizationalRole
cn: The Postmaster
newAttribute: newValue

The command where files filedst.ldif and filesrc.ldif are involved:

$ diffLDIF -s filesrc.ldif -t filedst.ldif

Output of diffLDIF command - delta.ldif

dn: cn=The Postmaster,dc=example,dc=com
changetype: modify
delete: oldAttr
oldAttr: attrValue1
-
delete: oldAttr
oldAttr: attrValue2
-
add: newAttribute
newAttribute: newValue

dn: cn=The Postmaster Remove,dc=example,dc=com
changetype: delete

ldifmodify

Replays the delta LDIF upon the content LDIF like the ldapmodify command.

If the diffLDIF is understood as the diff than the ldifmodify can be understood as the unix patch command.

Usage:

ldifmodify: Apply LDAP operations from LDIF to LDIF (like ldapmodify)

ldifmodify [FLAG] [LDIF Files for applying]

  -? --help[=FORMAT]  Show usage information (optional format)
  -V --version        Show version information
  -v --verbose        Higher verbosity
  -q --quiet          Lower verbosity
  -f --baseFile=FILE  Base LDIF File
  -o --outFile=FILE   Output LDIF File

Example is based on diffLDIF files

$ ldifmodify -f filesrc.ldif delta.ldif

The output of the ldifmodify example command above is filedst.ldif content.

The summary of the both diffLDIF and ldifmodify examples:

• diffLDIF: Create patch or diff between filesrc.ldif and filedst.ldif with name delta.ldif
• ldifmodify: Apply the patch file delta.ldif on filesrc.ldif and get the filedst.ldif

The above procedure is helpful when the delta.ldif is some kind of the patch applied on real LDAP directory, e.g. adding the users or updating of the structure.

ldif2html

It generates the brows-able LDIF as the HTML page.

Usage:

ldif2html <input.ldif> [<input2.ldif> <input3.ldif> ... <inputN.ldif> ] <output.html>

Download

• Darcs repository at http://patch-tag.com/r/dixiecko/ldif
• HackageDB Page at http://hackage.haskell.org/package/ldif
• Archlinux AUR at http://aur.archlinux.org/packages.php?ID=34060

Changes

• 0.0.6+7
  • added ldifmodify
  • changed interface of the library
• 0.0.5
  • fix in diffLDIF (missing new entries from second ldif not in first ldif - adds)
• 0.0.4
  • add ldif2html

Shallow Rampa - Message Board

Message Board

by dixie on Date unknown

Tagged as: chat.

Shallow Rampa - #Socky.ZU

#Socky.ZU

by dixie on Date unknown

Tagged as: chat.

About

Channel about the stuff which matters.

IRC Logs analysis results:

• The weekend talks are not often
• The most of the talks happen at the Monday
• The intensity of the talks decreases from Monday till Friday

Connection

• Preferred way http://webchat.quakenet.org/
• Or over pure IRC client (xchat, pidgin, irssi, epic4, bitchX…)

Shallow Rampa - Socky Clouds

Socky Clouds

by dixie on Date unknown

Tagged as: data.

About

Few visualizations of the IRC logs from #Socky.ZU channel. Images produced with wordcloud program from diagrams haskell package.

Size of the letters reflects the weight. The placement and colors are random.

Nicknames cloud

Weight is How many lines particular person wrote on the channel.

Nicks

Words cloud

Weight is How often particular word appeared on the channel.

Words

Sentences cloud

Weight is How often particular sentence appeared on the channel.

Sentences

Shallow Rampa - TCP Loop Connect

TCP Loop Connect

by dixie on Date unknown

Tagged as: network.

TCP state machine allows that TCP client establishes connection using connect() to itself. Look on diagram and focus on Simultaneous Open part.

Preconditions

Following conditions are needed for the loop:

1. Connection is established from HOST to HOST on the same IP
2. Destination TCP Port has to be from range of ephemeral ports configured on OS
3. Listen port is unbound

Probability

Preconditions don’t automatically guarantee the loop situation. But they are needed. Loop happens when OS bind the TCP port to client attempting to connect on the same. It can happen only for ephemeral ports.

The correctly designed TCP client/server application should not use the ephemeral ports for service side listener. Otherwise there is a chance it will establish unwanted connection to itself - loop.

In general it depends on various conditions like how OS assign ephemeral ports and how TCP client constructs sockets, but probability increases when:

1. TCP client creates for each connection new socket and OS assigns new ephemeral port (done usually in some high-level languages by programmers)

/* 
 * Try establish connection with 127.0.0.1:33333
 * THIS IS BAD EXAMPLE because it behaves like infinite loop at various 
 * conditions 
 */

do {  
    socket = getSockFromSystem();      /* it may use always different ephemeral port */
    connectTo(socket, 127.0.0.1, 33333);  /* it may connect to itself if ephemeral port == 33333 */ 
} while(isNotConnected(socket));

/* ... continue with established connection ... */

Ephemeral ports

For the details please read article describing them, basically on linux they can be displayed using:

$ cat /proc/sys/net/ipv4/ip_local_port_range 
32768	61000

Do not use for TCP lister port the value from that range.

Example TCP Client

Compile as gcc -o tcploop tcploop.c

/* tcploop.c */
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <assert.h>
#include <sys/socket.h>
#include <arpa/inet.h>
#include <stdlib.h>
#include <netdb.h>
#include <string.h>

int main(int argc, char *argv[])
{
  int port = 0;
  int sckfd = 0;
  int opt = 1;
  struct sockaddr_in *remote;
  char *ip = "127.0.0.1";
  int rc;
  long n = 0;

  if(argc != 2){
    printf("Usage: %s <listen port>\n", argv[0]);
    exit(1);
  }
  port = atoi(argv[1]);

  /* Set to Localhost:Port */
  remote = (struct sockaddr_in *)malloc(sizeof(struct sockaddr_in *));
  remote->sin_family = AF_INET;
  assert(inet_pton(AF_INET, ip, (void *)(&(remote->sin_addr.s_addr))) > 0);
  remote->sin_port = htons(port);

  /* create socket */
  assert((sckfd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP)) > 0);
  assert(setsockopt(sckfd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt)) >= 0);

  printf("Trying to connect..."); fflush(stdout);
  while(1) {
    n++;
    rc = connect(sckfd, (struct sockaddr *)remote, sizeof(struct sockaddr)); 
    if(rc < 0){
      if(n % 1000 == 0) { printf("."); fflush(stdout); }
      continue;
    }
    else {
      printf("Connected after %ld tries\n", n);
      break;
    }
  }
  /* Wait for Enter */
  printf("Press Enter to Continue...\n");
  getchar();
  close(sckfd);
  return 0;
}

Example output

Three commands outputs:

1. Show how Ephemeral port range is configured
2. Invocation of tcploop with port from that range (33333)
3. lsof output for port 33333 in the other terminal

$ cat /proc/sys/net/ipv4/ip_local_port_range 
32768	61000

$ ./tcploop 33333
Trying to connect..............Connected after 11263 tries
Press Enter to Continue...

$ lsof -n | grep 33333
COMMAND     PID  USER   FD      TYPE     DEVICE  SIZE/OFF       NODE NAME
tcploop    3440 dixie    3u     IPv4     163396       0t0        TCP 127.0.0.1:33333->127.0.0.1:33333 (ESTABLISHED)

Shallow Rampa - TreeMap Haskell library

TreeMap Haskell library

by dixie on Date unknown

Tagged as: data, app, lib, tree.

Overview

treemap-html is Haskell library and tools for construction foldable HTML tree view.

For the details about real Treemaps see Wikipedia Treemapping

To be honest the library doesn’t construct the real treemap with sizes of rectangles on the map based on some node weight parameters. It is only nested boxes tree view and package names are unfortunately misleading.

Installation

On Linux use HackageDB as usually.

On MS Windows you can use also binaries, see dirmap.exe and xml2treemap.exe.

$ cabal install treemap-html
$ cabal install treemap-html-tools

Please be aware that it is an experimental throw away software.

Usage of library

The function treeMap takes the Data.Tree with the String nodes and render it to the String with HTML elements.

Simple example which explain it:

import Text.HTML.TreeMap
import Data.Tree

exampleTree :: Tree String
exampleTree = Node "Slovakia" [
                 Node "Bratislava" [
                     Node "Petrzalka" [] 
                 ], 
                 Node "Zilina" [ 
                     Node "Hajik" [
                        Node "Mr. Cirman" [] 
                     ] 
                 ]
              ]

main = putStrLn (treeMap exampleTree)

Screen shot from Firefox (unfolded with clicks on the labels):

TreeMap Example

Usage of tools

Two basic commands provided:

1. dirmap
2. xml2treemap

dirmap

Creates tree view from the directory content.

Usage:

Usage: dirmap [OPTION...] directory
  -v          --verbose        verbose output on stderr
  -l Integer  --level=Integer  maxlevel for recursive directory listing
  -n Integer  --name=Integer   maximum size of filename. than truncation.
  -o FILE     --output=FILE    output FILE
  -f          --files          include files in listing
  -e                           do not included subtrees without files

Example usage:

$ dirmap -f -n 1000 -l 2 -o libs_l2.html /usr/lib

xml2treemap

Creates tree view from the XML.

Usage:

Usage: xml2treemap [OPTION...] <input file.xml>
  -v          --verbose       verbose output on stderr
  -o FILE     --output=FILE   output FILE

Example usage:

$ xml2treemap -o map.xml /usr/share/dbus-1/interfaces/org.kde.KWin.xml

Examples

The boxes are unfolded with mouse click on their text labels.

• Linux source code dirmap

Linux source code dirmap

• Hackage DB modules treemap

Hackage DB modules dirmap

• Ant Build XML with xml2treemap

Ant Build XML with xml2treemap

Download

• Binaries for MS Windows:
  • dirmap.exe
  • xml2treemap.exe

Sources

• Darcs repository
  • treemap-html http://patch-tag.com/r/dixiecko/treemap-html
  • treemap-html-tools http://patch-tag.com/r/dixiecko/treemap-html-tools
• HackageDB Page
  • treemap-html http://hackage.haskell.org/package/treemap-html
  • treemap-html-tools http://hackage.haskell.org/package/treemap-html-tools

Changes treemap-html

• 0.1 (later) marked as a depreciated package
• 0.1 initial release to hackage

Changes treemap-html-tools

• 0.2 (later) marked as a depreciated package
• 0.2 xml2treemap command included
• 0.1 initial release to hackage

Shallow Rampa - Wikipedia4epub

Wikipedia4epub

by dixie on Date unknown

Tagged as: ebook, app.

Overview

Wikipedia4epub is command line application which creates the offline ebook from articles on wikipedia.

It doesn’t provide the complete offline wikipedia can download given article and its direct children (wiki4e-mkepub-subtree)

Examples via wiki4e-mkepub-subtree:

• ebook from EPUB article
• ebook from Nintendo_Entertainment_System article

Installation

The best is to use HackageDB, the GHC 6.12.1 and newer is required.

$ cabal install wikipedia4epub

Please be aware that it is still ALPHA quality software.

Usage

There are following commands:

• wiki4e-mkepub-subtree http://en.wikipedia.org/<acticle>

Example of usage (wiki4e-mkepub-subtree)

$ wiki4e-mkepub-subtree http://en.wikipedia.org/wiki/EPUB
# STAGE 1/4 - Fetch starting article: http://en.wikipedia.org/wiki/EPUB
[1/1] Already cached. Skipping download. /home/dixie/.wiki4e/wiki4e_fetch/EPUB
# STAGE 2/4 - Fetch children articles: 113
[1/113] Already cached. Skipping download. /home/dixie/.wiki4e/wiki4e_fetch/EPUB
[2/113] Fetching : http://en.wikipedia.org/wiki/Filename_extension
...
[11/113] Fetching : http://en.wikipedia.org/wiki/DTBook
[12/113] Fetching : http://en.wikipedia.org/wiki/Website
...
[113/113] Already cached. Skipping download. /home/dixie/.wiki4e/wiki4e_fetch/Main_Page
# STAGE 3/4 - Sanitize articles
# STAGE 4/4 - Download images
Count = 352
[1/352] Already cached. Skipping download. /home/dixie/.wiki4e/wiki4e_images/100px-EBookreal.jpg
[2/352] Already cached. Skipping download. /home/dixie/.wiki4e/wiki4e_images/50px-Question_book-new.svg.png
...
Done.

For a performance reasons, the downloaded articles amd images are cached. Cache has to be cleanup manually.

For caching purpose following directories are constructed:

• $HOME/.wiki4e/wiki4e_fetch/
• $HOME/.wiki4e/wiki4e_sanitize/
• $HOME/.wiki4e/wiki4e_images/

Screenshot

The quality of screenshots is poor because they were shot with a phone camera.

Listing of the books in Reader

Book Listing. First is Wikipedia from Firefox

Context table of the Wikipedia articles

Context Table. Each Chapter is one Article

One selected Wikipedia article

Article Content, Top

The same article after scrolling

Article Content, Scrolled

Source code & bugs

Darcs repositories:

• Library for EPUB library: http://patch-tag.com/r/dixiecko/epub
• Wikipedia4epub program: http://darcsden.com/dixie/wikipedia4epub

For reporting the bug or questions please write me email to

Changes

• 0.0.12 <2011–11–11>
  • remove all commands except subtree
  • replace hardcoded usage of wikipedia url
  • thanks to previous point the book can be created from any wikipedia language
• 0.0.10 … 0.0.11
  • development amnesia
• 0.0.9 <2011–02–17>
  • decompress gzip compressed pages
  • use makeValid for sanitize of filenames
  • set stdout to binary mode
• 0.0.7 and 0.0.8
  • broken uploads
• 0.0.6 <2010–07–21>
  • enable relative hrefs within epub
  • minor fixes (allow the ’:’ char within name of the article)
• 0.0.5 <2010–03–14>
  • enable hrefs within articles in one epub
  • new command wiki4e-mkepub-subtree fetchs article and children
  • new command wiki4e-mkepub-cache makes epub from cached articles
  • known issue - space leaks during article hardvesting for images
• 0.0.4 <2010–02–13>
  • create temporary files at getAppUserDataDirectory()
  • display progress of the download
• 0.0.3 <2010–01–11>
  • download images

EPUB Format

It is standarized & open format for ebooks, but basically it is ZIP-ed XHTML pages, images with some metadata. For the details see EPUB Article on Wikipedia

It is suppored with my ebook reader (PRS–505) and also my new ebook reader HanLin V5.

The “full blown” open source software for ebook management - Calibre - supports the EPUB too.