Setting Up StrongSwan (IPSec) and OpenVPN

this is a mirror of the page at : http://jfcarter.net/~jimc/documents/vpn-setup-1501.html

James F. Carter <[email protected]>, 2015-01-23

I have recently reorganized my home network’s wild side naming and addressing, and the challenge now is to update the VPNs’ configurations and to get them working again. 

Network Names and Addresses 

For security by obscurity, in this document example.net is my domain name, certified by a reputable trust vendor (Startcom), whereas example.org is the internal name certified by my own self-signed X.509 root certificate. Although this root cert is installed for TLS on my personal machines, it’s a challenge to get Android to use it consistently, which is why I have the split naming scheme. Also, if I want to pass a URL to someone else, they will not have my root cert. 

There is one wild side interface which gets a dynamic IPv4 address from my ISP (Verizon FIOS). When it changes it is registered with my outsourced DNS vendor (dyn.com) under the name example.net (no 1-component hostname). It is also registered with my IPv6 tunnel broker (Hurricane Electric). The wild side IPv6 address is fixed. The internal fixed IPv4 address of the router is also registered in Dyn’s DNS under the name jacinth.example.net, and the same address is known to the internal DNS server as jacinth.example.org.

I have three VPNs: StrongSwan (IPSec), OpenVPN on port 1194/udp, and OpenVPN on 443/tcp. The latter is the last choice, but it is unfortunately very common for hotel Wi-Fi nets to block all ports except 53, 80 and 443 (TCP only). HTTPS service on example.net is provided on a nonstandard port; in fact I have a small collection of these: 

  • 1443 — OOBA, password authentication
  • 1444 — OOBA, X.509 client authentication
  • 1445 — OwnCloud and Roundcube (mail), auth by Kerberos, password, etc.
  • 1446 — OwnCloud and Roundcube (mail), X.509 client authentication
  • 1447 — Generic HTTPS

OOBA means Out of Band Authentication; it opens a firewall hole for the client, and without it no traffic gets in except for the VPNs, OOBA itself, and incoming mail. If the remote network is blocking ports, the OOBA ports definitely will be included. 

StrongSwan (IPSec) 

As with all VPN software, StrongSwan’s error messages are clear if you already know what the problem is; in other words, they are arcane if your configuration is messed up. Server’s /etc/ipsec.conf 

On the server (example.net), /etc/ipsec.conf reads like this. Remember that within a section each line must begin with whitespace including comments and otherwise blank lines.

config setup
    # Message verbosity for normal operation:
    charondebug = "dmn 0,mgr 1,ike 0,chd 1,job 1,cfg 0,knl 1,net 1,tls 1,lib 0,enc 0,tnc 0"
    # Verbosity for debugging problems:
    #OFF charondebug = "dmn 2,mgr 2,ike 2,chd 2,job 1,cfg 0,knl 1,net 1,tls 1,lib 1,enc 0,tnc 0"

conn %default
    auto = add                  # We're a responder, start when peer connects
    dpddelay = 0                # Rely on rekeying for dead peer detection
    dpdaction = clear
    
    left = %any
    leftauth = pubkey
    # See below about what this cert has to certify.  The root certificate
    # that signed it, and intermediate certs, must be in 
    # /etc/ipsec.d/cacerts/ (symbolic links OK).
    leftcert = /etc/ssl/hostcerts/host.crt
    leftsendcert = always
    leftdns = 192.168.200.193,2001:470:1f05:844::3
    
    right = %any
    rightid = %any
    rightauth = pubkey
    # Any client whose cert is signed by this CA is admitted. 
    # Use this command line to extract the Distinguished Name in your root
    # certificate in the format StrongSwan wants to see:
    # openssl x509 -in root.crt -noout -nameopt sname,sep_comma_plus_space -subject
    # The certificate, and any intermediate certs, must be in 
    # /etc/ipsec.d/cacerts/ (symbolic links OK).  
    rightca = "C=GN, L=Minas Tirith, CN=Example.Net Root Cert 2024"
    rightsendcert = ifasked
    # Assign the client's IP from these pool(s)
    rightsourceip = 192.168.200.160/29,2001:470:1f05:844::c8f0/125

conn roadwarrior
    # The peer just wants access to example.org
    leftsubnet = 192.168.200.192/26,2001:470:1f05:844::/64

conn defaultroute
    # The peer needs to send its default route down the tunnel
    leftsubnet = 0.0.0.0/1,128.0.0.0/1,::/1,8000::/1

Server’s /etc/ipsec.secrets

/etc/ipsec.secrets (readable only by root) needs a line for the host key corresponding to the leftcert. See man 5 ipsec.secrets for the format. If the key has a passphrase you can put it after the key’s filename. This means, if the black hats get onto your machine, in addition to stealing the host key they need to also steal /etc/ipsec.conf. This is a rather small increment in security and is a major hassle for other services that use this host key. The identifier should be (I’m pretty sure) the Common Name certified in leftcert (versus a SAN that the client relies on). 

jacinth.example.org : /etc/ssl/private/host.key

Server’s Host Certificate (Leftcert) 

The leftcert has to satisfy two conflicting requirements if you want to use IPSec on Android. You can designate one trusted CA cert, which Android StrongSwan is going to push to the server to induce it to trust the client cert that it also pushes. (The server doesn’t believe in this cert; it uses the cert to identify its own copy which it does trust.) But the same CA cert will be used to establish trust in the server’s host cert (leftcert) — which must certify the hostname that the client used to connect to the server. For the latter a SAN (Subject Alternate Name) is accepted.

I tried without success to use the host cert for example.net that was certified by Startcom: if the Android client had the Startcom cert it refused to send the client cert (certified by example.org, not Startcom); if it had the example.org cert it rejected the server’s host cert signed by Startcom; and if it was told to select a CA cert automatically, it sent cert requests for about 200 CA’s, and I’m not sure what it selected, but trust was not established.

I finally realized that just because Startcom has certified example.net, doesn’t mean that I can’t certify it too. I created a new host cert for jacinth.example.org (Common Name) with a SAN for example.net (and some aliases that I need). Now the Android client believes in its own client cert and in the server’s host cert.Android Configuration

You first need to load your client certificate and key into Android’s certificate storage. Your best bet is to obtain from your Certificate Authority a PKCS#12 file (extension .p12 or .pfx) containing your key, cert, intermediate CA cert(s) (if any) and root cert. It will most likely end up in /sdcard/Download or whatever alias Android is using this year. Start the Settings app, find Security, scroll almost to the bottom and find Credential Storage, and under that, Install from SD Card. In the file listbox click on Download, and you should find your downloaded PKCS#12 file. Click on it, give the password, give a friendly name for the content (which Android will not show when it’s most important), and you’re done. 

To edit the VPN profile, start StrongSwan and long-press on the line item for your connection; the headline changes and you hit Edit. Or for a new connection, just hit Add VPN Profile. On the profile page: 

  • The profile name is arbitrary, just a descriptive title (short and useful).
  • The gateway is the server host. Give a hostname (or IP address if fixed) which you can actually resolve and connect to. It must be certified by the leftcert that the server will send you. 
  • Type is IKEv2 Certificate. There are other possibilities, including a loginID and password (EAP), which I am not using. 
  • User Certificate: You would pre-load your client cert into Android’s certificate storage, and now you select it from the User collection. 
  • CA Certificate: I was able to connect using either the root certificate, or the intermediate certificate that signed both the client cert and the server’s host cert. Using the root cert is more sanitary. 
  • Hit Save when you have finished editing. 

Turning On IPSec (Android)

To turn on IPSec, short-click on the line item for your connection. It should connect promptly. In case of problems a message box will pop up with a choice to view the log file. Linux (Network Manager) Configuration

The Network Manager icon is in your toolbar. On Wi-Fi the icon is the traditional signal strength bars; for a wired connection it is a picture of two computer monitors. Left click to get the connection menu. Near the bottom is a line for VPN Connections; slide to the right and a submenu will open, whose second from last item is Configure VPN. Click on it to get a list of VPN definitions. Click on one (IPSec) and hit Edit, or hit Add. Give the root password (twice) if in paranoia mode. Fill out non-VPN tabs according to your normal policy, specifically on the General tab, I mark All users may connect to this network. On the VPN tab for IPSec: 

  • Gateway Address: Give the server’s hostname or (fixed) IP. Pick one that you can resolve and connect to. It must be certified by the host certificate that the server will send you. 
  • Gateway Certificate: This is your trusted copy of the root cert that signed the server’s host cert, in PEM format. The label is not very specific about what it wants. If there are intermediate certs, they and the root cert should be concatenated.
  • Client Authentication: Certificate/Private Key. There are alternatives that I’m not using, such as a loginID and password (EAP).
  • Client Certificate: This is your own certificate. Network Manager is not as picky as Android; it can be signed by a different CA than the Gateway Certificate.
  • Private Key: The key that goes with the client certificate. See the discussion above about unencrypted private keys. If there is a passphrase you will be asked for it. 
  • Options: I do request an inner IP address (on the server’s net), I don’t insist on UDP encapsulation (StrongSwan will turn this on if ESP isn’t going to work), and I decline compression to resist the BEAST attack. 
  • Hit Save when finished. 

Turning On IPSec (Network Manager)

To turn on IPSec, click on the Network Manager icon, slide to the right end of VPN Connections, and click on the menu item for IPSec. It just takes a few seconds to initiate the connection, if it’s going to work. A padlock is added to the Network Manager icon. If it fails, look in /var/log/debug for clues (assuming one configured /var/log/debug).

OpenVPN (Ports 1194 and 443) 

The configuration for the two OpenVPN ports is almost identical and I will describe both together. On Android the app is OpenVPN Settings by Friedrich Schäuffelhut, and the binary program that it downloads and installs is OpenVPN-2.1.1. Current in OpenSuSE 13.1 dated 2014-12-01 is OpenVPN-2.3.2.Server’s /etc/openvpn/server.conf

The server configuration file goes like this. Host-specific parameters are grouped at the end. 

# Verbosity:
verb 1
mute 10

# Preserve root-only files and options.
persist-key
persist-local-ip
persist-remote-ip
persist-tun

# Lock key and buffers in memory, keeping them out of the swap file.
mlock

# Use a dynamic tun device.  (Could also be tap, for ether bridging.)
dev tun

# Should we use DF for path MTU discovery?  Empirically verify the MTU?
mtu-disc maybe
mtu-test

# Dead peer detection by pings
keepalive 15 31
ping-timer-rem

# Don't complain if started when the network isn't up yet.
ifconfig-nowarn

# Resist denial of service attacks.
connect-freq 1 1

# Allow reconnects with a different IP address (DHCP renew does that sometimes)
float

# Allow multiple connections from the same user, e.g. from different hosts.
duplicate-cn

# https://wiki.debian.org/OpenVPN recommends to push a DNS server for Android.
push "dhcp-option DNS 192.168.200.193"

# Crypto Parameters (must match the peer, can't push them)
# HMAC algorithm (anti-tampering checksum)
auth SHA256

# Cryptographic cipher on main data channel (not used in tls-server/client mode)
cipher AES-256-CBC

# Use LZO compression (with adaptive shutoff)
comp-lzo

# Polarity of this host (tls-client or tls-server)
tls-server

# Diffie-Hellman parameter file, only on server.  
# You should generate your own; runtime: 13 sec on Intel i7-3632QM @ 2.2GHz
# openssl genpkey -genparam -algorithm DH -out dh2048.pem -pkeyopt dh_paramgen_prime_len:2048
dh /etc/openvpn/dh2048.pem

# Server-specific options:
# Protocol and port
proto udp
port 1194
# proto tcp
# port 443

# Multi-client server, uses dynamic addresses from 192.168.200.128/28,
# 16 addresses, 4 per client and the server takes 1 set.  A different
# address range is used for the port 443/tcp server.  
mode server
server 192.168.200.128 255.255.255.240
max-clients 3

# To get on, the client must present a certificate signed by a CA in 
# this file.  PEM format.  Multiple certs may be concatenated.  Include
# intermediate certs.  
ca /etc/ssl/ca/example.org.crt

# If a different root certificate signed the server's host cert, list it
# (and intermediate certs) here or append to the cert file.  
# extra-certs /etc/ssl/ca/example.net.crt

# The server's host certificate and private key (unencrypted).  Recommended
# to appeend the intermediate cert(s) and trust anchor that signed it.
cert /etc/ssl/hostcerts/host.cia
key /etc/ssl/private/host.key


Android OpenVPN Configuration File

On Android the authentic OpenVPN binary is used so the configuration file is nearly identical. It differs in these aspects:

# Accept configuration overrides from the server
pull

# Slightly different ping/keepalive parameters:
ping 60
ping-exit 180
ping-timer-rem

# Server's hostname or IPv(4 or 6).  Use a name you can resolve and connect to.
remote example.net

# Require this Common Name in the certificate which the server will send over.
# Too modern: verify-x509-name jacinth.example.net name
# This option is deprecated:
tls-remote jacinth.example.net

# Polarity of this host (tls-client or tls-server)
tls-client

# Unlike on real Linux, the certificates and key go in the OpenVPN directory
# /sdcard/openvpn , and are specified by relative paths. 
# This is the CA cert(s) that signed the server's host cert.  PEM format,
# and include the intermediate cert(s) if any.  
ca example.net.pth

# This is the CA cert(s) that signed our client cert. 
# extra-certs example.org.pth

# The client's user certificate and private key (unencrypted).
# You are allowed to concatenate the root and intermediate certs and
# to omit extra-certs.  
cert example.org.cia
key example.org.key

You also need to set some preferences. Long-press on the line item for the connection and from the menu pick Preferences. 

  • Use VPN DNS Server (turn on)
  • VPN DNS Server (click on it, and fill in the server’s IP address)
  • Enable Logging (turn on). Now there will be an item in the long-press menu for View Log File. It is rewritten on every connection, and for successful connections it’s not too verbose. 

An issue with OpenVPN is, the tunnel cannot go through itself; there has to be a route from the client to the gateway’s wild side for the tunnel packets to follow. But payload packets to the gateway’s wild side will follow the same route, not through the tunnel. If inimical forces are blocking my payload packets, they will continue to do so with OpenVPN. Or if you have sensitive information not protected by TLS (I don’t), OpenVPN will not be protecting it either. The cure for that is to connect to payload services on the internal address (jacinth.example.net), which willgo through the tunnel.Turning On OpenVPN (Android)

To turn on OpenVPN, launch the OpenVPN Settings app. The first menu item is for turning on the whole OpenVPN mechanism. Then short-click on the line item for your connection. It should connect promptly with progress notes below the connection title. In case of problems turn it off, then long-click on it and pick the choice to view the log file.Linux (Network Manager) Configuration

The Network Manager icon is in your toolbar. On Wi-Fi the icon is the traditional signal strength bars; for a wired connection it is a picture of two computer monitors. Left click to get the connection menu. Near the bottom is a line for VPN Connections; slide to the right and a submenu will open, whose second from last item is Configure VPN. Click it to get a list of VPN definitions. Click on one (OpenVPN) and hit Edit, or hit Add. Give the root password (twice) if in paranoia mode. Fill out non-VPN tabs according to your normal policy, specifically on the General tab, I mark All users may connect to this network. On the VPN tab for OpenVPN: 

  • Gateway Address: Give the server’s hostname or (fixed) IP. Pick one that you can resolve and connect to. It must be certified by the host certificate that the server will send you. 
  • Authentication Type: Certificates (TLS). Password authentication is also possible. 
  • User Certificate: This is your own certificate. Network Manager is not as picky as Android; it can be signed by a different CA than the Gateway (CA) Certificate.
  • CA Certificate: This is your trusted copy of the root cert that signed the server’s host cert, in PEM format. The label is not very specific about what it wants. If there are intermediate certs, they and the root cert should be concatenated.
  • Private Key: The key that goes with the user certificate. See the discussion above about unencrypted private keys. If there is a passphrase, fill it in the next text box. 
  • Advanced – General: Use LZO compression (must match the server’s choice).
  • Advanced – Security: the cipher and HMAC need to match the server’s configuration; they cannot be pushed from the server because the control channel (encrypted) has to be established before anything can be pushed.
  • Advanced – TLS Authentication: Subject Match = jacinth.example.net. This is the Common Name in the server’s certificate. The GUI shows an example where /CN= is prepended, but I believe I tried it and it didn’t work; I never found out why. I leave the other items turned off. 
  • Hit Save when finished. 

Turning On OpenVPN (Network Manager)

To turn on OpenVPN, click on the Network Manager icon, slide to the right end of VPN Connections, and click on the menu item for OpenVPN (normal or tls/443). It just takes a few seconds to initiate the connection, if it’s going to work. A padlock is added to the Network Manager icon. If it fails, look in /var/log/debug for clues (assuming one configured /var/log/debug).

Testing the VPNs 

These hosts were tested as clients: 

  • Selen: Samsung Galaxy S-3 (cellphone) running CyanogenMod-11-M12 based on Android-4.4.4 KitKat. It is directly using cellular data on the wild side.
  • Mica: Asus Transformer Pad Infinity (tablet) running CyanogenMod-11-M12 based on Android-4.4.4 KitKat. It communicates on the wild side via Selen’s Wi-Fi Hotspot (hostapd).
  • Xena: Sony Vaio SVS1512ACXS (laptop) running OpenSuSE 13.1 (Linux). It also communicates on the wild side via Selen’s Wi-Fi Hotspot (hostapd).

My firewall rules prevent many of these tests from working unless the payload packets go through the VPN tunnel. These tests may or may not work without the VPN: 

  • You can do traceroute to an outside host. If the first hop is/isn’t on the VPN gateway, that proves that the tunnel was/wasn’t in use.
  • DNS for [host.]example.net should work whether or not the tunnel is used. But example.org is only available through the tunnel. 
  • Connections to https://$host:1443 or 1444 (OOBA service) will work from the wild side. Look at the reported IP address to see if the tunnel was used. But connections to other ports will hang and time out. Also the gateway’s IPv6 address may be tried and will time out.
  • Firefox has a feature that if you connect to http(s)://example.net/ and it fails, including timeout, Firefox will retry on http(s)://www.example.net/ . In my case this is a CNAME to jacinth.example.net, which has the internal address so traffic will go through the tunnel and connect successfully. Nonetheless, when this behavior is noticed it should count as a failure. 

On Android, the Hurricane Electric Network Tools app was used for the DNS and ping tests, whereas on desktop Linux dig and/or host was used. Firefox was used to test URLs on both OS’s.

On Android when you use cellular data DHCP will give you the IPv4 addresses of your ISP’s DNS server(s). These will not give service to outside hosts, specifically to packets coming from your VPN gateway. Therefore you need to change the DNS server. On Android-4.2 Super Jelly Bean and earlier, you would do setprop net.dns1 8.8.8.8 (Google’s free DNS service is shown). However, starting in 4.3 or 4.4 KitKat DNS queries are directed to netd, a local caching nameserver (which some forum posters say is there to prevent ad blockers from working). There is a new API to control who netd forwards to. And Android-4.4.3 and earlier has a bug in this API, preventing DNS alteration apps from controlling netd. Fortunately, CyanogenMod-11-M8 and later (2014-06-xx) is based on Android-4.4.4 which has the bugfix. 

IPSec on Android does not obey the DNS server announced over the VPN. To get the right DNS server with IPSec I’m using the DNS Forwarder app by Evan He (free, ad supported), which has presets for many popular recursive DNS services; you can also configure your own custom server. It requires root access. For these tests I made changes in this order: 

  • With the VPN off, use the ISP’s DNS or forward to Google.
  • Turn on the VPN. It needs working DNS to resolve the gateway’s IP.
  • Change forwarding to the internal (example.org) DNS server, accessible only through the tunnel. 
  • When tests are done, change forwarding back to Google or to the ISP. 
  • Turn off the VPN.
StrongSwan 
(IPSec)
OpenVPN 
(1194/udp)
OpenVPN 
(443/tcp)
Test Selen Mica Xena Selen Mica Xena Selen Mica Xena 
How long to connect2s 15s2s 4s 5s 6s 8s 10s14s 
DNS for example.net [1]ok okok[5]okok ok[5]okok ok[5]
DNS for example.org [1]ok ok ok[5]okok ok[5]okok ok[5]
Ping to internal IPv4 adrok ok ok ok ok ok ok ok ok
Ping to internal IPv6 adrFAILFAILFAILFAILFAILFAILFAILFAILFAIL
Ping to wild side IPv4 adrok ok ok [3][3][3][3][3][3]
Ping to example.net (wild side)ok ok ok [3][3][3][3][3][3]
Ping to jacinth.example.net (internal)ok ok ok ok ok ok ok ok ok
Traceroute to arachne.math.ucla.edu [2]ok ok ok ok ok ok ok ok ok
http://jacinth.example.net/ ok[4]okok ok ok ok ok ok ok
https://jacinth.example.net:$PORT/ok[4]okok ok ok ok ok ok ok 
http://example.net/ ok[4]okok [3][3][3][3][3][3]
https://example.net:$PORT/ok[4]ok ok [3][3][3][3][3][3]
  • [1] OK means it got the SOA, A, AAAA and MX records.
  • [2] OK means the first hop was on the gateway, proving that the tunnel was being used; the packets were not going direct.
  • [3] Packets are not expected to go through the tunnel; this should not count as a failure. 
  • [4] Usually it asks for the IPv6 address first, promptly asks for IPv4, and sometimes but not always it hangs for about 30 secs. Then usually it retries DNS and is able to connect and show the page with no further delays. 
  • [5] I have a custom /etc/resolv.conf and it had to be hand-edited to use the DNS server for example.org. 

Conclusion on testing: all three VPNs are fully functional except for these deficiencies: 

  • All the VPN servers announce which DNS server should be used (the one in example.org), but none of the clients obey this option. Android’s OpenVPN client has a feature to override DNS, which works. For IPSec on Android you need to use a separate app to override DNS. 
  • On Xena a custom resolv.conf is used which does not automatically use the DHCP provided DNS server, whether correct or not. 
  • Selen’s Wi-Fi hotspot (hostapd) never passes through IPv6 traffic.
  • OpenVPN is unable to pass IPv6 traffic through the tunnel. There are hints that this may be functional in the current version, but it is not set up at present. 
  • Selen itself is unaware that it could send IPv6 traffic through the IPSec tunnel. Thus IPv6 is non-functional for three different reasons.
  • Firefox on Selen with IPSec attempts IPv6 and has about a 30 second timeout before it retries with IPv4, successfully. 

Issues to be worked on in the future: 

  • See if OpenVPN will now pass IPv6 through the tunnel. 
  • See if automatic DNS switching can be made to work on Xena. 
  • Re-test IPv6 on a network natively capable of IPv6.
  • It might be wise to use separate IPv6 addresses for the internal and wild side gateway interfaces. 

The extend of ad-tracking

I had setup Pi-hole, a remote one, DNScrypt, both local and remote, privoxy for cleaning up the bad web traffic that passes through pi-hole etc. Things were looking good and that’s when EFF came up with their new https://panopticlick.eff.org/

After all the effort, the Panopticlick reports are not shining with colors. This gives and idea about the extend to which tracking is prevalent.

The funny part is, this is what you you get with all the circus !!

Useful links & screen shots:

https://github.com/pi-hole/pi-hole/wiki/DNSCrypt-2.0

https://github.com/pi-hole/pi-hole/wiki/DNSCrypt-2.0

https://itchy.nl/raspberry-pi-3-with-openvpn-pihole-dnscrypt

[email protected]:~# apt install dnscrypt-proxy

no privacy – as logs are there !

General DNS pointing to one location
randomized !

Unique Software Training initiative

I was looking at the possibility of staying at an Himalayan Village for a while and working. Interestingly, I stumbled up a training firm called Alt-Campus : https://altcampus.io/

Clockwise from top: Skyline of Dharamsala, Main Street Temple – McLeod Ganj, Gyuto Karmapa, Himachal Pradesh Cricket Association Stadium and St. John church
(Image from https://commons.wikimedia.org/wiki/File:Dharamsala_Montage.png)

Yes, the cool thing is that they have setup this up in Dharmasala!

They seem to have an hands on training which could be useful. The payments are only after one get placed and that sounds pretty cool thing to do as well.

I will post more details, if I find anything new …. !

Featured image from : https://en.wikipedia.org/wiki/Dharamshala#/media/File:Cloudy_Triund,_above_Mcleod_Ganj,_Himachal_Pradesh.jpg

NodeJS Performance questionnaire

As someone from a C++ – Java – UNIX background Javascript on the server side was always a peculiar thing for me. For the current application https://streamersedge.com/ dApp combining blockchain, streaming, games and much more NodeJS has become a very important tool chain.

For the performance, a questionnaire of the sort is prepared. Publishing it here as I would like to save them and also get some critics from the experts.

The eventual plan is to use Prometheus, Grafite, Grafana stack for Metrics and ELK (ElasticSearch, Logstack, KibanaO for log aggregation. We are also experimenting with some APMs like Newrelic in the meantime.

NodeJS VM Garbage collection
  1. What is the GC (garbage collection) parameters we have ?
  2. How heap size/RAM is available for the NodeJS process ?
  3. Are we using all the CPUs available on a given server or instance ?
  4. What are the sysctl and process security limits for the NodeJS processes in a given server/instance ?
  5. Have we optimized the network connections to the servers to support maximum connections ?
  6. For given CPU and RAM, say, 1 Ghz and 1 GB RAM, roughly how many concurrent connections we can support ?
  7. Are we running I/O bound NodeJS processes ? (This is when DevOps will show your BPF super powers ? :slightly_smiling_face: )
  8. OpenTracing – Can we use Grafana – ELK to map the CPU spikes ?
  9. Newrelic /Dynatrace or an Open Source solution for APM ?

What is stored in a blockchain ?

information!

Understanding the type of information stored in the blockchain will help us to decide how to implement Oracles means for  the cross chain communication etc.

A blockchain is a Distributed OpenLedger of Records Contracts & Transactions between multiple parties.

Records = Assets, People, In Game assets, Non Fungible Assets, a real world farm.


Contracts = A transfer of an asset (say, a farm) between 2 people can be depicted in a contract. In the real world a contract will be defined in a legal language. Various laws of the state will become applicable. In the context of a blockchain, a contract will be written in a “Smart Contract Language.” Blockchains that support contracts have their own language constructs applicable only in their respective context.


Transactions = The action of transferring an asset, or an action on a unique address or transfer of value is defined as a transaction in the context of a blockchain.

Kerala House Boats – Engine, Kitchen etc.

From Sun's makeup box | സൂര്യ ദേവന്റെ കണ്ണാടി.

House Boats, Aleppey, Kerala

 

Needless to say, the house boats of Kerala, to be precise in Alleppey is quite popular and attracts lots of tourists. I was curious about the the engine and other finer details, I could find in a trip and here it is:

 

The hidden engine room door in the kitchen

The hidden engine room door in the kitchen of the house boat

Engine

Engine

 

from the engine room

turning the engine - kerala house boat

turning the engine – kerala house boat

electrical wiring, plumbing and septic tank

electrical wiring, plumbing and septic tank

 

 

 

me fishing!

Its been sometime since I wanted to try my luck with fishing again – that means, during my lower primary school days I used tail along Kurichya friends in Wayanad and they used to let me cast the “choonda” with earth worm bait and once or twice gave me their knife with which I hit the water and nothing happened. So, Yesterday, we decided that enough is enough & we will get choonda again. Armed with earth worms, we bought that stick with hooks, went to nearby backwaters, alas in 10 minutes we got 2 fishes.

Now all inspired we went back and bough a bigger stick, oh, fishing raad. Then the class began. The guy at the shop is a champ with this stick ! Poor me. I just wanted to catch fish and now I have this graphite stick, “flow”, a rotating whatever for the “line” with 3 bearings, swivel (2 level one to be specific), clip / lock, I took the lessons well. Oh yea, got few led balls as weight too.

Oh the most important part is “Loore” – this fish like thing is to fool fishes. People claim that the fishes are foolish enough to hit sorry, “bait harder or faster”. I think I already know how to cast this thing – thanks to the “mooppan” who had the patience to teach me this.

After paying for this stick, which was a hefty sum, went to the earlier spot from where we got 2 fishes, successfully casted in the second try, casted again, again, finally realised that the fishes are not foolish enough to be “lured” into. And earth worms may work better. Or may be the fishes are foolish and they don’t know to deal with sophisticated sticks.

Never knew that this fishing thing is a hobby where people talk about the size of their “sticks”, weight it can handle, swivel, clip, lock, line, flow – these seems to be status symbols. (almost like people flaunting their cars, FX cameras, 1.2f 50mm lenses blah blah). Anyways, here I am with 2 fishing rods, one which can catch fishes with live baits, ie earth worms and another super sophisticated stick with even the fishes not understand.

If anyone in an around kochi knows where these ultra sophisticated fishes who understands, loore and other stuff, please do let me know !

Aadhar aka UID project – questions and concerns.

There is confusion and ambiguity about the Unique Identity aka Aadhar project implemented in India by UIDAI. This post is an attempt to compile the questions and concerns from various sources. I will update the post with clarifications when available & try to keep updated with recent developments. My personal opinions or ideas will not be added & will try to ignore anything that sounds illogical. Credit to authors will be given at all times when its possible to find the authors & this post is under Attribution-ShareAlike 2.5 India for India & under Attribution-ShareAlike 3.0 Unported in all other countries.

Concerns:

Here are some questions on Aadhaar which not answered at al1. The UID was promoted as a `voluntary’ `entitlement’. Now, people are being threatened that they canot access any services or institutions unless they are enrolled for a UID. This has not happened only because states have so decided, but are based largely on the reports of various committees.


2. This coercion has been introduced into a project where no feasibility study has been done till date. On 28 September, 2010, 17 eminent citizens had raised questions about the launch of a project of this nature without even a report on feasibility. The Parliamentary Standing Committee had raised similar concerns. There are no answers yet,and yet the coercion has become the norm.

3. The Standing Committee on Finance roundly rejected the National Identification Authority of India Bill 2010, by its report presented to Parliament on December 13, 2011. It also said that the UID project needs to be sent back to the drawing board, citing various deficiencies in its plan and execution.

The UIDAI and government have chosen to ignore the report. There is still no law. And the UID project, and the compulsion that has been introduced, are occurring beyond the protection of the law.

4. Proof of Concept studies were done only after the project was already underway. While doing the PoC on enrollment (uploaded in February 2011, over 4 months after enrollment had begun), the report says, they did not include arecanut workers and other plantation workers because this would only complicate the sample. How can such a study find validity. Also, the findings of the study are not substantiated; the evidence is not available, and those reading the report critically have found it to be self-serving.

The Fingerprint Authentication report (March 2012) and the Iris Authentication report (September 2012) suffer from similar problems. And unreasonable assumptions have been used to make it appear like the technology can be made to work: for instance, it is said in the report that iris never changes! A study by two professors at Notre Dame demonstrates that this is in fact untrue, and that those who have been saying it have done so because no longitudinal studies had been done thus far, since this is such recent technology!

5. The UIDAI, is pushing for `re-engineering’ all systems and `seeding’ all data bases with the UID number to make it `ubiquitous’. This will make all systems dependent on the UID system functioning. Already, the inability to authenticate people is showing signs of requiring the use of `manual override’, which is a magnificent source of `leakage’ and of exclusion of those the systems does not authenticate. This unseemly haste to shift to untested and undependable technology is a source of grave concern.

6. The seeding of the number everywhere also raises privacy concerns. UIDAI has not denied that the UID project raises serious privacy issues. But UDAI has sidestepped the issue, saying that it is an issue wider than the UID project and needs a general law, and so he will not worry himself about it.

Whatever his assumption of responsibility to ensure the protection of privacy — and in this case it would include concerns of profiling, tracking, tagging, convergence of data, data mining, the state and all manner of people gaining access to this data bank that is being created especially in the context of data becoming a transactable commodity — must precede consideration of any such project. Instead, there is no law governing the project and the uses to it which it may put; and there is no law on privacy either.

7. While still on privacy, it is being propagated that `the poor have no use for privacy’. This casual dismissal of such a right, especially given how vulnerable the poor, including classes of migrants, homeless, jhuggi dwellers, casualised workforce, for instance, are needs to be confronted.

8. There is much concern about the companies to whom the UIDAI has given contracts. There are companies like L1 Identity Solutions whose favoured customer has been the CIA, where a former director of the CIA, George Tenet, was even on their Board. Accenture Securities Ltd, another company shortlisted for the project, is on Smart Borders Project with the US Homeland Security. US law requires all agencies to provide any information demanded of them to the Homeland Security if asked.

When the UIDAI was sent an RTI asking why they had enlisted foreign companies such as these in the project, the answer was that they had no way of knowing whether they were foreign companies — because the way invited participation did not elicit this information!

The absence of a law means that there is nothing binding them to a legal structure within which we could hold them.
The contracts with these companies are being denied for public perusal in the name of `confidentiality’

9. The rampant outsourcing in the project means that all manner of people handle our data, including biometric data, and there is little that we can do when it is traded, misused, shared, lost ….

In January 2011, the Home Ministry had said that they could not use UIDAI data because it was insecure, unverified and could pose a security threat. Then they patched up their differences and decided to share the country 50:50! As citizens, though, the rapprochement does not answer the problems raised by the Home Ministry.

Where are the answers?

 

Confusion between NPR & UID projects.

 

It seems there is confusion regarding the purpose two identical projects and no one seems to know the difference.

 

Court orders requesting clarification:

 

Ownership of data

It seems there is no clear answer as to who owns the data and recent news articles like “Your data, going on sale soon” discusses about these ownership issues.

Safety of data collected

There are numerous incidents of data collected being found discarded. The biggest such reported event is this one : Maharashtra loses data of 3 lakh UID cards

 

 

What is the difference between a Bill and an Act ?

 

Legislative proposals are brought before either house of the Parliament of India in the form of a bill. A bill is the draft of a legislative proposal, which, when passed by both houses of Parliament and assented to by the President, becomes an Act of Parliament. As soon as the bill has been framed, it has to be published in the news papers and suggestions are invited from the general people, and after going through the suggestions of the people the bill is amended and then Bill may be introduced in the Parliament by ministers or private members. The former are called government bills and the latter, private members’ bills. Bills may also be classified as public bills and private bills. A public bill is one referring to a matter applying to the public in general, whereas a private bill relates to a particular person or corporation or institution. The Orphanages and Charitable Homes Bill or the Muslim Waqfs Bills are examples of private bills. – wikipedia.org

US Patriot Act & security of Indian citizen data

In a nutshell, the act can force companies registered in the United States or their subsidiaries elsewhere to share any or all data to law enforcement agencies. A detailed description can be found here: USA Patriot Act

This is a major issue for many technology companies including Google, Apple, Microsoft, Facebook, Yahoo, Salesforce, Data.com etc. The act is in clear violation of EU data protection regulation, UK privacy act which clearly states individual data can be stored only in UK or safer harbors but access must not be allowed. Swiss regulations makes this more complicated. As far as India is concerned IT act 2000 and its 2008 amendments  and not talking about any of these issues. The privacy law is still under scrutiny. There are incidents of many firms deciding against doing business with US based companies of which some are listed below.

  1. European Firm Refuses To Go On the Microsoft Cloud Due to PATRIOT Act Concerns
  2. Microsoft admits Patriot Act can access EU-based cloud data
  3. PATRIOT Act and privacy laws take a bite out of US cloud business

Now lets back to our context. This simply means that other countries can access all our biometric data without any permission whatsoever from us or our government. If one thinks, this is OK, then this discussion is definitely for such people.

The question has been raised here long back but I am not sure how and why we are going to circumvent it. Here is a discussion about it which even activists seem to be not noticing – The Trouble With Big Brother’s Eye

 

To be added:

Time period of validity of a bill, Reasons for a bill not be presented in Parliament, Is there a provision under RTI to find out why a bill is not presented to the parliament.

 

Photo editing with Freedom – the GIMP way

 

Until recently, the issues with FOSS and photo post processing was loss of data while RAW conversion, poor colour management, lack of 16  bit support etc.

Finally it seems Gimp 2.9 is addressing the 16 – bit issue which was until recently addressed only by CinePaint. Yes there are multitude of options like Photivo, Krita etc but IMHO if all these efforts are combined in building a better GIMP, RAW support etc things could have been better. Looks like too much of a freedom is not always the right way unless there is a guiding force.

Gimp 2.9  30 May 2013 development branch

Gimp 2.9 30 May 2013 development branch with a unified / single window, support for images with upto 32 bit floating point precision (hurray!!), better colour profile management – trust me! my TIFF images looks the same in Mac OS X & GNU/Linux now, GEGL

Here I am trying to get Gimp 2.9 working on GNU/Linux flavours and if possible on Mac OS X as well & explaining photo processing with Gimp 2.9. I am thinking of using RawTherapee (http://rawtherapee.com/) for RAW conversion as I am not happy with UFRaw. For the colour management I am using Open-ICC profiles, Argyll CMS & UI

The workflow:

  1. Photos are taken in Adobe RGB 1998
  2. Monitor uses calibrated profile
  3. Processing is done with ProPhoto profile from OpenICC bundle
  4. RAWTherapee is used for RAW conversion
  5. Final output in JPEG and TIFF are saved with Adobe RGB 1998
  6. Gimp 2.9 compiled from source is used as it supports 16 – bit image formats
  7. RAM is used for GEGEL swap for faster performance by mounting tmpfs with 6GB (instead of default 50% physical memory size)
  8. Linux SWAP is also getting used upto 2.8 GB

 

Setting the stage

The first step is to install OpenICC profiles

Compile Gimp 2.9 as per the instructions

1. http://crunchbang.org/forums/viewtopic.php?id=25716

2. http://www.gimp.org/source/howtos/gimp-git-build.html

Install RAW therapee & get an image ready.

 

Errors & fixes:

The first error I came across was babl accessing a different process and can be fixed with a less “secure” kernel setting as shown below.

Change ptrace_scop by Editing  /etc/sysctl.d/10-ptrace.conf & the resultant value should be: kernel.yama.ptrace_scope = 0

Keel in mind that this will be applicable only after restart & we can change proc values as follows.

# echo 0 >  /proc/sys/kernel/yama/ptrace_scope

 

Second error:

Yet to find a solution for this.

babl-memory.c:114 babl_malloc()

(script-fu:1976): LibGimpBase-WARNING **: script-fu: gimp_wire_read(): error

 

 

I will be updating the document with more insights and automation. For now, the above steps should work and may be you want to look into some GIMP tutorials like: http://blog.patdavid.net/2013/03/the-open-source-portrait-postprocessing.html

 

Updated: 30-05-2013