Jumping Bean Blogs

Alfresco Share Email Documents Action

2025-05-10T11:54:24Z

When we moved from Drupal to Liferay back in 2020, sadly we didn't have time to migrate our old blogs to the new platform. In our logs, we still see hits for these old URLs, and we have decided to make a concerted effort to resurrect these old blogs for those who find them useful. For most of the popular blogs, we are managing to get the original article content back and repost, predominantly unmodified except for URLs and other minor changes for the migration. So please bear in mind the content may now be a little dated, especially since the articles were already old when we migrated.

Unfortunately we cannot recover all of the blog posts. This is one of them. The post on the time when we extended Alfresco Share to add an action to be able to directly email documents from Alfresco Share, rather than having to download them and then attach them to emails, is one of them. Although we still use Alfresco, Alfresco Share is now deprecated but still supported, and effort is being put into the Alfresco Developer Framework. Can't say I will miss Surf.

Contact Us if you need any Alfresco development work or consulting.

Anyway, for those looking for this article, all we can currently provide is the link to the GitHub repo. We hope you find it useful.

Alfresco Email Documents Share (GitHub)

Published: [Original Publication Date Unknown, Likely Prior to 2020]

JPA 2 Criteria API Tutorial

2025-04-24T05:37:06Z

Submitted by Mark Clarke on Sun, 07/10/2011 - 14:00

When JPA 2 was released in 2009 it included the new criteria API. The purpose of the API was to get away from using JQL strings , (JPA Query Language), in your code. Although JQL seems like a great way to leverage your existing SQL knowledge ,in the OO world it has a major drawback namely; there is no compile time checking of your query strings. The first time you find out about a spelling or syntactical error in your query string is at run time. This can be quiet a productivity drain with developers having to correct, compile and redeploy to continue.

Unit testing your code goes some way to addressing this problem but one area that cannot be addressed by unit tests is refactoring. Most refactoring tools battle with strings and you are stuck with rerunning unit tests and correcting each string that slip through the manual changes on each iteration of the test until all is well. Now with the JPA criteria API it's possible to have type safe queries that are checked at compile time and refactoring is much more efficient!

Get Expert Java Training

Java: Elevate your programming with our fundamental Java Training courses.
Spring: Unlock enterprise Java skills with our focused Spring Framework training

JPA Criteria API

There is a price to pay for this static checking though. First you need to generate a set of meta model classes, more on what these are is given below, that describe the fields of your entities; luckily this is easily achieved by placing a step in your maven build process. The second price you pay is verbosity and a less than intuitive-at-first-glance api.

Generating the JPA Criteria Meta Model Classes

To generate the meta model classes during your build process add the following plugin details to your maven pom.xml. In my case, because I have a different persistence unit for our unit tests I had to specify which persistence unit I wanted built, otherwise a bug causes the plugin to throw an error when it finds an entity listed in two persistence units. (i.e the production config and test unit config). I make use of Eclipselink in the pom.xml below.

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-source-plugin</artifactId>
</plugin>
<plugin>
    <groupId>org.bsc.maven</groupId>
    <artifactId>maven-processor-plugin</artifactId>
    <version>1.3.5</version>
    <executions>
        <execution>
            <id>process</id>
            <goals>
                <goal>process</goal>
            </goals>
            <phase>generate-sources</phase>
            <configuration>
                <compilerArguments>-Aeclipselink.persistencexml=src/main/resources/META-INF/persistence.xml -Aeclipselink.persistenceunits=mypu</compilerArguments>
                <processors>
                    <processor>org.eclipse.persistence.internal.jpa.modelgen.CanonicalModelProcessor</processor>
                </processors>
            </configuration>
        </execution>
    </executions>
</plugin>

Now, at compile time, the set of meta model classes should be automatically generated for you.

How to use the JPA Critiera API

The criteria api can seem quiet daunting at first but it's not that bad once you grok its basic design approach. There are two main object that you will use to create your query, namely the CriteriaBuilder object and a CriteriaQuery object. The first step is to get a handle to a CriteriaBuilder object and then create a CriteriaQuery object. This is is done with the following boiler plate code, where em is an EntityManager object.

CriteriaBuilder cb = em.getCriteriaBuilder();
CriteriaQuery cqry = em.createQuery();

From the CriteriaQuery object you will create the four "components" of a query namely:

Criteria Components

Component	Description	How to create
Select	The objects or object fields you wish to return. Pretty much like the select part of a JQL query. You can do aggregations here too and return fields from objects but in this brief tutorial we will just select objects.	CriteriaQuery.select
From	This is where we stipulate which entity (table) we are quering. You can also follow relationships to other entities that are part of the root entity i.e "joins" and also subSelects.	CriteriaQuery.from
Where	This is the where you stipulate the criteria you wish to apply to the entities that you are selecting. You create "Predicates" which are used to build up the "where" clause.	CriteriaQuery.where CriteriaBuilder.equals CriteriaBuilder.lessThanOEquals etc.
Order By	If you wish to stipulate the order object should be returned in, you do it here	CriteriaQuery.orderBy
Group By	For aggregations you stipulate the object fields or objects here.	CriteriaQuery.groupBy

In practise nearly every method from the CriteriaQuery and CriteriaBuilder takes an object that implements the interface java.persistence.criteria.Expression, Selection or Predicate, which can be very unhelpful when deciding what to send into the method for a beginner; especially since a lot of the objects implement both interfaces and the Expression interface inherits from the Selection interface! (Does that make your brain hurt? I am sure the API desinger must have had an anuerism).

It really doesn't make sense that a Path object can be sent to the where method. But it best not to pay too much attention to these high level interfaces and understand the process of creating a CritieriaQuery, at least in the beginning.

The simplest approach to understanding the API is to separate out the task of creating a query into the following steps:

Create your CriteriaBuilder and CriteriaQuery objects,
Setup your from clause,
Setup your select clause,
Setup your criterias or predicates
Setup the where clause using your predicates
Execute the query.

A Simple CriteriaQuery Example

Lets assume we have a simple object called MyEntity which has the following fields:

dateCreated - Date, and
age - Integer

Here is a simple CriteiraQuery example.

             //Boilerplate

             CriteriaBuilder cb = em.getCriteriaBuilder(); //Step 1
             CriteriaQuery cqry = em.createQuery(); //Step 1

             //Interesting stuff happens here

             Root<MyEntity> root = cqry.from(MyEntity.class); //Step 2
             cqry.select(root); Step 3
             
             //Boilerplate code
             Query qry = em.createQuery(cqry); //Step 6
             List<MyEnity> results = qry.getResultList(); //Step 6

So we have two boilerplate sections that you need to type up, section 1 and section 6. Section 1 creates the criteria objects you need and the section 6 , at the end, executes the query.

The main work happens in the middle, steps 2 - 5, where you construct your query. The simple example above will return all entities in the table mapped to the MyEntity class. We had to tell the query which entity we were selecting from with the "from" method, and then what we wanted returned from the query with the "select" method.

Using our step-by-step approach we can begin to build more complicated queries. Lets say we want to use some criteria in the query. To do this we need to populate the "where" method of the CriteiraQuery with "Predicate" objects.

A predicate object is usually created by using one of the methods on the CritieraBuilder class, although the Expression interface also creates some Predicates, but for now just think of using the CriteriaBuilder class for this task. So lets say we want to find all MyEntity objects with age > 10.

            Root<MyEntity> root = cqry.from(MyEntity.class); //Step 2

            cqry.select(root); //Step 3
            Predicate pGtAge = cb.gt(root.get("age"),10); //Step 4
            cqry.where(pGtAge); //Step 5

When creating a predicate we have to tell the API which field of which object we are comparing against. We may have more than one entity that we are quering across, as with a join for example, so you need a reference to the entity being queried.

The object returned by the "from" method above is what we use here. We then use the "get" method to stipulate which field from the entity object we want to compare against. Yes. the API is sadly very verbose.

If we wanted to string more than one criteria together , lets say age >10 and dateCreated>"2011-07-01" then we would create two Predicates and "and" them as follows:

       //assume we have created a date object for 2011-07-01 
       //called date
 
             Root<MyEnity> root = cqry.from(MyEntity.class); //Step 2
             cqry.select(root); //Step 3
             Predicate pGtAge = cb.gt(root.get("age"),10); //Step 4
             Predicate pGtDateCreated=
                      cb.greaterThan(root.get("dateCreated"),date); //Step 4
             Predicate pAnd = cb.and(pGtDateCreated,pGtAge); //Step 4
 

             cqry.where(pAnd); //Step 5

Using Meta Model Classes in Query

Using the many methods on CriteriaBuilder you can build up sophisticated "where" clauses. Use the autocomplete on your IDE to see what methods are available or check out the API docs.

You may be wondering why we are using strings in our "Predicate" objects. Afterall doesn't this defeat the purpose of not using JQL? Are CriteriaQueries subject to the same failings then as JQL queries that we setout in the introduction?

The answer to that question is to use the Meta Model Classes that we created in the beginning of this tutorial. Once the classes have been generated you can refer to fields in Entity objects using the meta model classes. The meta model class has the same name as your Entity class with an underscore(_) appeneded. So to say we are comparing against the age field we would use MyEntity_.age. The query above is rewritten below using the meta model classes.

            //assume we have created a date object for 2011-07-01 
            //called date
             Root<MyEnity> root = cqry.from(MyEntity.class); //Step 2
             cqry.select(root); //Step 3
             Predicate pGtAge = cb.gt(root.get(MyEntity_.age),10); //Step 4
             Predicate pGtDateCreated=
               cb.greaterThan(root.get(MyEntity_.dateCreated),date); //Step 4
             Predicate pAnd = cb.and(pGtDateCreated,pGtAge); //Step 4
             cqry.where(pAnd); //Step 5

Criteria Query Using Joins

So what is we want to query across entities i.e do a "join" query? Lets say we have another Entity object called AnotherEntity with fields as follows:

name - String
enabled - boolean

In addition we extend the MyEntity object to have a reference to AnotherEntity object. So MyEntity becomes:

dateCreated - Date,
age - Integer and
anotherEntity - AnotherEntity

Now lets say we want to query for all MyEntity objects that have an AnotherEntity object which are disabled. We would do this as follows:

            Root<MyEnity> root = cqry.from(MyEntity.class); //Step 2
            Join<MyEntity,AnotherEntity> join =
                             root.join(MyEntity_.anotherEntity); //Step 2
            //Join<MyEntity,AnotherEntity> join =
                             root.join("anotherEntity"); //Step 2
 


            cqry.select(root); //Step 3
            Predicate pGtAge = cb.gt(root.get(MyEntity_.age),10); //Step 4
            Predicate pGtDateCreated=
                             cb.greaterThan(root.get(MyEntity_.dateCreated),date); //Step 4
            Predicate pEqEnabled = cb.equals(join.get(AnotherEntity_.enabled),false);
            Predicate pAnd = cb.and(pGtDateCreated,pGtAge,pEqEnabled); //Step 4
             cqry.where(pAnd); //Step 5

You can see that our form step is getting more complex as we stipulate what to join on. As with the Predicates one can use string or the meta model classses to stipulate the desired field to join on. If you going through the hassle of using the criteria API then there really is no point in using strings!

More Complex Select Clause

We will now look at more complex select clauses, our step 2. Lets say we are only interested in the dateCreated field of our MyEntity object. Our code would look something like:

                 Root<MyEnity> root = cqry.from(MyEntity.class); //Step 2
             cqry.select(root.get(MyEntity_.dateCreated)); //Step 3

If we wanted to use an aggregate function and get the minimum dateCreated we could use something like:

             Root<MyEnity> root = cqry.from(MyEntity.class); //Step 2
             Expression min =
                      cb.min(root.get(MyEntity_.dateCreated));//Step3
             cqry.select(min); //Step 3

There are method for returning array of objects or tuples from a query and also a way to create new object on the fly from fields or queries entities. I may cover these in a future tutorial but for now this should be enough to get you going on the Criteria API.

As you start using the Criteria API from JPA2 you will see that there are other ways of creating or manipulating some of the "components" we set out above other than following the step-by-step approach, but if you ever get lost just follow the steps.

IPv6 - Set Up An IPv6 LAN with Linux

2025-04-22T06:00:24Z

IPv6 - Set Up An IPv6 LAN with Linux

Submitted by Mark Clarke on Sun, 04/05/2015 - 18:05
This post has been recovered from our old blog. A lot has changed since 2015.

Setting up an IPv6 LAN with Linux? Ever wondered how to do that? For years we have heard the dire predictions about the impending doom of IPv4 and the imminent arrival of IPv6. As with any eschatological prediction you either choose to ignore it and hope for the best, or you prepare for the event as best one can. So far the former strategy has served many sys admins well and proved to be an effective strategy .

If however, you have decided to gird your loins and face IPv6 head on, you probably quickly discovered, that, although there is a lot out there about the theory of IPv6, there is very little in the way of practical how tos when it comes to setting up an IPv6 LAN.

What makes understanding IPv6 troublesome is the complexity of working in a mixed environment of IPv4 and IPv6. This complexity becomes evident when one tries to connect to an external IPv6 network or the Internet which is still predominantly IPv4.

Get Linux Certified. Get Ahead.

Hands-on LPI training and Linux Foundation courses

Steps to Set Up an IPv6 LAN

This blog posts break this down into two separate problems:

A - Setting up an IPv6 LAN network with Linux,
B - Connecting your IPv6 network to the Internet

If you separate these two issues out its much easier to figure out what you need to do. Both of these steps have issues that need to be understood before the IPv6 "ah-hah" moment. Once that happens you will also have the "oh no" moment which might help you understand why there is such slow movement on IPv6 adoption.

In the first part we will configure an Ubuntu 14.10 server to manage an IPv6 LAN. In the 2nd part we will deal with the myriad of options to connect an IPv6 network to the internet.

IPv6 Addressing - Some Theory

First we need to cover some theory on IPv6 addresses. There is a lot of article covering IPv6 addressing on the web, so I will just summarize what you need to know to proceed with setting up an IPv6 network. There are some nuances and subtleties we will brush over to provide you with a working conceptual model.

IPv6 addresses consist of 8 groups of 16 bit hexadecimal numbers to give a total address of 128 bits. (See Global addresses below for explanation of the 2001:0db8::/32 address block.)
- 2001:0db8:85a3:0000:0000:8a2e:0370:7334
- 2001:db8:85a3:0:0:8a2e:370:7334 -> leading 0 are dropped & groups of zeros (0000) become 0,
- 2001:db8:85a3::8a2e:370:7334 -> lastly consecutive zeros are replaced with an empty double colon ::
The first 4 group of hexadecimal numbers of an address, 64 bits of the 128 bits, is the network prefix (network mask). All IPv6 networks have a 64 bit network prefix,
The remaining 64 bits are the host identifier,

Sometime you will see an addresses listed with a prefix such as /48 or /56 etc. This does not mean that 16 (64-48) or 8 (64-56) bits of the 64 network prefix has been reserved for use by hosts as with IPv4 CIDR. The network address is always 64 bits long.

This notation refers to a block of networks. i.e all networks that begin with the first 48 or 56 bits set as specified. This is known as a routing prefix and is used in routing rules, resulting in smaller routing tables. It is also used for when you are assigned a block of IPv6 networks.

The idea with IPv6 is that you should be assigned a block of networks by your ISP or IANA instead of a single host address or single IPv4 network as currently happens with IPv4.

The remaining bits of the network prefix 16 (64-48) or 8 (64-56) are called the subnet id. So the routing prefix + subnet id make up the network prefix of an IPv6 address. Dont' be confused by the use of the word subnet in subnet id.It is not an IPv4 subnet mask. It is simple the part of the network prefix you get to assign yourself as the administrator of that block of network addresses.
So if you get an IPv6 address block with a 56 bit routing prefix it means you can have 255 (28) networks each with 1.844674407×10¹⁹ (264) hosts!. Its up to you to determine how the subnet portion is used to create the network address. So if you are given a block of IPv6 network such as fdc8:282a:f54c::/48 it means you can have 216 networks. Your networks addresses are :

fdc8:282a:f54c:1::/64
fdc8:282a:f54c:2:/64
...
fdc8:282a:f54c:ffff:/64

We wil come to the host identifier portion later later. The IPv6 network address space has been "sliced up" into different blocks. What you need to know about these blocks is given below: (each address block is explained further later)

Special IPv6 Address Blocks

Name	Prefix	Explanation
Link Local	fe80::/10	Although this routing prefix is only 10 bits leaving 54 bits for up to 254 networks only one subnet id has been allocated so far by the specification which is fe80:0:0:0 or fe80::/64
Unique Local Addresses(ULA)	fc00::/7	Although this routing prefix is only 7 bits, the 8th bit must always be 1 according to the spec. So what you will see in practice is fd00::/7. At some later point we may see fc00::/7. We will be using this address block in our setup.
Global Addresses	2001::/23	Global addresses will in fact be most of the address space of IPv6 So far the 2001::/23 block has been assinged and this is what you are likely to see in practice until further blocks are assigned to regional registrars. Within this some addresses have been reserved for a special purpose such as 2001:0db8::/32 which is reserved for documentation so if anyone copies it it won't actually route. To see what block have been assigned see the IANA site.

For more information on the address blocks see the IANA site

A - Set Up an IPv6 Network

1) Set Up an Link Local Only IPv6 LAN with Linux

We will set up an IPv6 network incrementally. We will start with the simplest and most trivial IPv6 and add services as we go. This will help us arrive at a understanding of how the various services fit together. We will go from the simplest IPv6 network to one which has all the basic network services required of a business network.

Simplest IPv6 Network - Link Local Only

To setup the simplest IPv6 network you just have to boot up a host or two with a IPv6 enabled operating system such as Ubuntu. Open a terminal and type:

"ip -6 address list"

You should see output similar to the following:

        1: lo:  mtu 65536 
        inet6 ::1/128 scope host 
        valid_lft forever preferred_lft forever 
        2: eth0:  mtu 1500 
        qlen 1000 inet6 fe80::922b:34ff:fe7b:6ff1/64 scope link 
        valid_lft forever preferred_lft forever,multicast,up,lower_up>,up,lower_up>

IPv6 link local addresses have been assigned automatically to any interfaces that you have. The IPv6 localhost address (IPv4 127.0.0.1) is ::1/128. You can do the same on another host to gets it IPv6 link local address and then do a IPv6 ping with "ping6" - note the 6.

        ping6 fe80::922b:34ff:fe7b:6ff1

The fe80::/64 network prefix is the link local network as explained in the table above. It should be the only IPv6 network address you will see across different physical networks. In fact every host on an IPv6 network must have an link local address (fe80::/64).

Host Identifier Generation

The host identifier portion of the link local address, the remaining 64 bits, is generated from the mac address with a algorithm applied to extend the 48 bit mac address to the 64 bit host address required for IPv6. See EUI64 for the algorithm used. The host identifier may also be manually assigned by the system administrator. This introduces the risk of duplicate IP addresses being assigned, so IPv6 has a duplicate address detection protocol that allows hosts to determine if there is a conflict before assigning itself an address.

In most cases you will let this be automatically generated. In IPv4 initially IP address had to be manually assigned or assigned via a DHCP server. Later the 169.254/16 address range was reserved for auto-configuration in IPv4 network. Unlike IPv4 your interfaces will always have an fe80::64 address, it is not used instead of a valid IPv6 address. In IPv6 your interfaces will typically have multiple IP addresses.

Why do you need a Link Local Address?

IPv6 configuration is done using layer 3 (network layer) protocols and not layer 2 (media layer eg. Ethernet) as with IPv4; so a valid IPv6 address is required before any additional configuration can be done. Of couese it also allows for zero config simple networks.

Pros and Cons of Link Local Network

With a link local address you can communicate with other IPv6 hosts on the local network segment or broadcast domain. i.e the same switch or shared media network. So for a home LAN not connected to the internet this is all that is required. You can connect to your printer, Smart TV, PlayStation etc automatically using protocols such as UPnP and multicast DNS (ZeroConf). Connecting to the internet, or a network in a different physical network or logical network, will require a bit more work.

2) Set Up A Stateless Routable IPv6 Network

If this all there was too it then we could all go home. But if you start to think about it you will begin to have some doubts as to how useful a link local only IPv6 network is.

How do I assigned the same address to a host every time without doing it manually? (it is possible for a nodes host identifier to change between reboot if there is a conflict)
What if I change the NIC and get a different IP address?
How do I configure the hosts for routes and other services such as DNS, NTP etc?
How do I communicate between two internal networks separated by a router or WAN link?

To address these issues you need to assign yourself a non-site local address. This can be a unique local address (ula) or a global address. For a global address you will need to get an IPv6 network block from your ISP or get one assigned to you by IANA. So we will make use of a ULA address which you can assign yourself.

What is the difference between a ULA and a Global address?

By convention a ULA is not routed over the public internet. Routers on the public IPv6 network should refuse to route such traffic in a similar manner to private IPv4 addresses. Essentially there should be no routing entries in the routers responsible for internet traffic, making them unreachable from outside an organisation.

If you are going to start experimenting with IPv6 there are two reasons to use a ULA

you should start with a ULA address to avoid any mis-configuration disasters.
It might be hard to get a global IPv6 address assigned to you. There are very few ISP handing out IPv6 network addresses currently so in some cases its the only choice available to you.

Unique Local Addresses

One feature of unique local addresses is that they should be different for every network you see. Unlike IPv4 where private addresses (196.128/16, 10/8 and 172.16/16) meant there are often networks with the same network mask - eg nearly every home and office has a network with the network mask of 192.168.1.0/24 address or 10.0.0.0/24 range; you might never see a duplicate IPv6 ULA network address. This is because only the first 8 bits of the network prefix are fixed at "fd". The remaining 56 bits of the netowrk prefix, the subnet id, can be randomly selected. System administrators are meant to create the subnet id themselves. A handy way to generate the subnet id for the ULA is to use a site like unique-local-ipv6.com. From here you will get a /48 address range meaning you can have up to 65356 private networks!

Its generally a good idea to use a random subnet id rather than generate one like fd01:1:1:1::0/64 as this increase your chance of a conflict. Why would you be worried about a conflict if these are not routable? Have you ever had to merge two network that had the same IPv4 address range? Have you ever tried to setup a VPN between two network with the same IP network range?

Global Addresses

Global address will be assigned to you by an ISP unless you get your own block and tell your ISP to route it to you. So much like you get a public IP address from your ISP for IPv4 you will in future, get an IPv6 network address range when you dial up. Note: not a single IP address but a whole block of IPv6 addresses. Depending on your ISP you may get only one network or be assigned a block with multiple networks. In this case the router will received the network address prefix to use on your network. It will work the same as for the steps below except instead of a ULA network it will be a global address. Note you don't get assigned a full IPv6 address. You get the network prefix.

So to summarise. You will need at least two IPv6 addresses for each interface if you want to do normal networkng tasks like route between network. A link local which is always present and at least one ULA or global address or perhaps all three!

For our exercise we will use ULA addresses to setup an IPv6 only LAN.

Set Up an ULA IPv6 Network in Linux

Note:You can setup the following IPv6 network on a network that is already configured for IPv4. You can run IPv6 in parallel with IPv4. This is known as a dual stack setup. Once done you can stop the IPv4 services and run the network on IPv6 only or keep it dual stack. One reason to test without IPv4 infrastructure running is to convince yourself that your network really is working over IPv6.

Ok, now things start to get a bit more complicated. First we need some more theory :( We have already seen how a link local address is assigned. But how is the ULA address assigned? The nodes will need some way to get the network prefix? For this IPv6 makes use of a router advertisement service that runs on the local network router. This is what they mean when they say SLAAC (Stateless Automatic Address Configuration) configuration. The process is as follows:

An link local address is generated by the host,
The host will ask (solicit) any routers for configuration information.
The router response with a router advertisement. This advertisement contains the network prefix the host should use and the address of the router for the default route.
The router may also provide a DNS address
The node generates the host identifier portion of the IPv6 ULA address and assigns it to the interface. (Note: The router does not provide the entire IPv6 address)

So a node now has an ULA IPv6 address and a default gateway and all should be good. This is known as stateless address assignment. The router does not assign an address per se. It has no idea what IPv6 address the hosts ends up using, only that it is in the provided network. Hence the stateless in the term stateless automatic address configuration (SLAAC).

Steps to Configure the Router Advertisement Service

The advertisement service can run on any Linux box, but that box will become the default route for IPv6 traffic. In future your ADSL router will provide router advertisement services. First assign the Linux box a static IPv6 address from the ULA network: (In the examples that follow I use the fd5f:12c9:2201::/48 ULA routing prefix and I have chosen fd5f:12c9:2201:1::/64 as the network prefix. (ie :1 is the subnet id).

Configure a static IPv6 on Ubuntu

        sudo vi /etc/network/interfaces

        auto eth0
        iface eth0 inet6 static
          address fd5d:12c9:2201:1::1
          netmask 64
          autoconf 0
          dad-attempts 0
          accept_ra 0

Now we need to install the router advertisement service:

Router Advertisement Daemon Configuration

sudo apt-get install radvd

vi /etc/radvd.conf

        interface eth0
        {
            AdvSendAdvert on;
            prefix fd5d:12c9:2201:1::1/64 {
                AdvOnLink on;
                AdvAutonomous on;
            };
            #Send DNS Server setting - assumes there is a DNS server setup at the address below
            RDNSS fd5d:12c9:2201:1::2{
            };
        };

Restart the service and then on a client restart the network. You should see two IPv6 address on your network card.

        ip -6 address list

You can ping the router with the ping6 utility:

"ping6 fd5d:12c9:2201:1::1" if this doesn't work try "ping6 fd5d:12c9:2201:1::1 -I eth0" -> Use the interface with the assigned IPv6 address. We will cover DNS and IPv6 in the net section.

Congratulations you have an IPv6 network up and running! If your router is multi honed and has two interfaces with IPv6 addresses you will be able to route between the two networks. You will need to setup two static IPv6 addresses in /etc/network/interfaces.

3) Set Up a Stateful ULA IPV6 LAN with DHCP & DNS Services

After the initial excitement of getting a routable IPv6 network up and running you start to realise you need a few more services:

What if I want to send down other configuration information such as the NTP or SMTP server settings?
What if I want to make sure the same IPv6 address, not just the network prefix, always get assigned to a server like the NTP or SMTP server?
What if I want to track IP address assignment?
What if I want to provide dynamic updates to the local DNS server for local hosts? After all remembering all these 128 bit random addresses is going to be hard.
How do I assing a IPv6 address in the DNS server zone file?

Sadly the router advertisement service can only provide a network prefix, default route and dns server address and not much else. To provide the required service we need to use a DHCP server. To get the node to use a DHCP server we need to configure the radvd service to tell nodes to contact a DHCP server for additional configuration information.

You can configure radvd to tell the nodes to contact the DHCP server for

configuration info only (stateless) or
for configuration information and its IP address (stateful)

So a DHCP server can be used in a stateless and stateful IPv6 setup. We will use DHCP to send configuration information such as DNS servers and to assign IP addresses since we want to assign fixed IP to well known hosts.

Edit the /etc/radvd.conf file

        interface eth0
        {
             AdvSendAdvert on;
             prefix fd5d:12c9:2201:1::1/64 {
                AdvOnLink on;
                AdvAutonomous on;
                AdvManagedFlag on; # get a full IP address from the DHCP server
                AdvOtherConfigFlag on; # get other configuration info from the DHCP server
            };
        };

Setting up DHCP6 is similar to DHCP for IPv4. We will use the isc-dhcpd-server:

"sudo apt-get install isc-dhcp-server"

Edit the /etc/dhcpd/dhcpd6.conf: (note comments removed for clarity)

        ddns-update-style interim;
        ddns-updates on;
        
        update-conflict-detection false;
        update-optimization false;
        
        option domain-name "jumpingbean.co.za";
        option dhcp6.name-servers fd5d:12c9:2201:1::2;
        
        default-lease-time 600;
        max-lease-time 7200;
        include "/etc/dhcp/rndc.key";
        
        log-facility local7;
        
        zone jumpingbean.co.za. {
                                  primary 127.0.0.1;
                                  key rndc-key;
        }
        
        
        zone 1.0.0.0.1.0.2.2.c.9.2.1.d.5.d.f {
                primary 127.0.0.1;
                key rndc-key;
        }
        
        subnet6 fd5d:12c9:2201:1::/64 {
             range6 fd5d:12c9:2201:1::100 fd5d:12c9:2201:1::200;
        }

A lot of the configuration for DHCPv6 is the same as DHCPv4 and I am assuming you are familiar with the configuraiton of DHCPv4 for dynamic DNS updates. Here we set up the DHCP server to

provide the DNS server
dynamically update the bind DNS server by specifying which Zone file should be updated
the domain name to use on nodes.
we could push additional routes, NTP server settings etc.

Note: To setup a fixed IPv6 address in DHCPv6 you make use of a DUID (Device Unique ID) which is not the mac address which is used for IPv4 DHCP. The DUID is assigned by the operating system and remains the same even if network cards change. The complete unique id is made up of the DUID and an IAID (interface assigned id) as you may have more than one interface on a node requesting IP addresses from the DHCP server. I.E Each interface has a unique ID.

Below is an example of a fixed assignment

        host example {
          host-identifier option dhcp6.client-id 31:30:30:30:30:31:33;
          fixed-address6 fd5d:12c9:2201:1::101;
        }

I am not aware of a way to get the DUID and IAD of a interface on a node in Linux. You can look in the leases file on the DHCP server once an address is assigned but this is no ideal. A binary copy of the node's DUID can be found at /var/lib/dhcpv6/dhcp6s_duid but it cannot be simple cat'ed to stdout. If anyone knows how to read the unique id for an interface on a node please let the internet know :)

DHCP Complexities

Before moving on to configuring the name server there are some issues to note about the DHCP server.

You can run an IPv4 and IPv6 DHCP server in parallel on the same network as they listen on different ports. So you can have a dual stack DHCP IPv6 and IPv4 network but it does require two running instances of the DHCP server.
To start the isc-dhcp-server in IPv6 mode you need to provide it with the option "-6". You can set this in /etc/defaults/isc-dhcpd-server via "OPTIONS="-6" on Ubuntu. BUT! on Ubuntu 14.10 this is ignored and the init v script will stubbornly start in IPv4 mode. To get the dhcp server to start in DHCPv6 mode add this to the /etc/rc.local file as a temporary solution and disable the init system from starting up the DHCP server.

sudo update-rc.d dhcpd disable

dhcpd -6 -cf /etc/dhcp/dhcpd6.conf -lf /var/lib/dhcp/dhcpd6.leases eth0

If you going to run dual stack you will nee to add this for the IPv4 mode of the DHCP server

dhcpd -4 -cf /etc/dhcp/dhcpd4.conf -lf /var/lib/dhcp/dhcpd4.leases eth0

You might also have apparmour prevent writing to the lease file if you try and write it to a different location. You can either stop apparmor or configure the dhcp server to write the lease file to location that its profile supports writing to.

DNS Set Up Bind9

First you will need to install bind9. DNS differs from DHCP in that only one instance of the DNS server needs to be running to service both IPv4 and IPv6 requests. It simple need to bind to both IPv4 and IPv6 addresses. A DNS server will provide answers to queries for an IPv4 or IPv6 address no matter what the source address of the query. i.e it doesn't look at whether the query came from an IPv4 or IPv6 address it just looks at what it was asked to lookup - the record type.

sudo apt-get install bind9

Next we need to edit the bind configuration to setup our zones and tell bind to listen on IPv6 or IPv6 and IPv4 interfaces. Most of this is standard bind9 setup as for IPv4. The only real change at this point is the AAAA records in the zone file and the awful reverse zone name that is a mission to type without making a mistake.

/etc/bind/named.conf.options

        options {
                  directory "/var/cache/bind";
                  forwarders {
                                8.8.8.8;
                  };
                  dnssec-validation auto;
        
                  auth-nxdomain no; # conform to RFC1035
                  listen-on-v6 { any; };
        };

/etc/named/named.conf.local

        zone "jumpingbean.co.za" {
                type master;
                allow-update { key rndc-key; };
                file "/var/lib/bind/jumpingbean.co.za";
        };
        
        zone "1.0.0.0.1.0.2.2.9.c.2.1.d.5.d.f.ip6.arpa" {
                type master;
                file "/var/lib/bind/fd5d:129c:2201:1";
                allow-update { key rndc-key; };
        };

The zone files below already contain some dynamically updated IPv6 address from the DHCP server. Those are the ones annotated with the TXT entries. You will only make entries for the static IPv6 addresses and let DHCP handle the dynamic address entries.

Zone files

/var/lib/bind/jumpingbean.co.za

        $ORIGIN .
        
        $TTL 604800     ; 1 week
        jumpingbean.co.za           IN SOA  ns.jumpingbean.co.za. no.jumpingbean.co.za. (
                                        182        ; serial
                                        604800     ; refresh (1 week)
                                        86400      ; retry (1 day)
                                        2419200    ; expire (4 weeks)
                                        604800     ; minimum (1 week)
                                        )
                                NS      ns.jumpingbean.co.za.
                                A       127.0.0.1
                                AAAA    ::1
        $TTL 300        ; 5 minutes
        android-a74e95670198fd6a A      10.0.10.4
                                TXT     "0002ec64161ce51591018b9eb0a01ae6b9"
        $TTL 604800     ; 1 week
        gateway                 AAAA    fd5d:12c9:2201:1::2
        ns                      AAAA    fd5d:12c9:2201:1::2
        $TTL 300        ; 5 minutes
        trinity                 A       10.0.10.3
        $TTL 187        ; 3 minutes 7 seconds
            
                            TXT     "025c83d7b0b5ca62d26381f057fbeed483"

/var/lib/bind/fd5d:129c:2201:1

        $ORIGIN .
        $TTL 604800     ; 1 week
        jumpingbean.co.za           IN SOA  jumpingbean.co.za. no.jumpingbean.co.za. (
                                        182        ; serial
                                        604800     ; refresh (1 week)
                                        86400      ; retry (1 day)
                                        2419200    ; expire (4 weeks)
                                        604800     ; minimum (1 week)
                                        )
                                NS      ns.jumpingbean.co.za.
                                A       127.0.0.1
                                AAAA    ::1
        $ORIGIN jumpingbean.co.za.
        $TTL 300        ; 5 minutes
        android-a74e95670198fd6a A      10.0.10.4
                                TXT     "0002ec64161ce51591018b9eb0a01ae6b9"
        $TTL 604800     ; 1 week
        gateway                 AAAA    fd5d:12c9:2201:1::2
        ns                      AAAA    fd5d:12c9:2201:1::2
        $TTL 300        ; 5 minutes
        trinity                 A       10.0.10.3
        $TTL 187        ; 3 minutes 7 seconds
                                TXT     "025c83d7b0b5ca62d26381f057fbeed483"

Now we have a fully functional IPv6 LAN. You may have IPv4 running at the same time and a interface can have a IPv4 and IPv6 address. To test DNS try the following (you can try it on public DNS servers too)

dig @DNS-IP www.google.com A

dig @DNS-IP www.google.com AAAA

You can query the local DNS server you just setup as well for an address you assigned.

So now you have a fully functioning IPv6 network. You can disable the IPv4 network and see that you can still reach all machines and services. On the client side you will not have to do any explicit configuration changes. Everything will be fine until you try and access the internet. The following will fail:

"ping6 www.google.com"

"ping www.google.com"

The first ping to google IPv6 address will fail unless you have global address. If you have an fd00::/7 address you won't get a response. The second will fail because you are trying to access an IPv4 host from an IPv6 network. And this leads to the second issue we need to deal with to understand IPv6.

B: Connecting to the Internet from an IPv6 Network

This is the most troublesome part of the IPv6 protocol. There are a myriad of "transition mechanism" designed to ease the transition form IPv4 to IPv6. None of them are ideal as far as I can tell and most have requirements which place them outside of the reach of small and medium businesses until IPv6 becomes more widespread.

At the time of writing this article the most common occurrence is for an ISP to assign a single IPv4 address to your router. As IPv6 becomes more widely adopted this will change and the step below may not be that relevant. The issue is that IPv6 services cannot be access via an IPv4 address and IPv4 services cannot be natively connected from an IPv6 only host. We are going to look at two solutions:

NAT64,
Using a tunnel broker

NAT64 Transition Mechanism

For the sake of simplicity we will assume an IPv4 address on the router. The advantage of NAT64 is that you can use IPv6 internally and still use your ISP public IPv4 address. Another advatnage is you can keep using your IPTables IPv4 firewall rules. The disadvantage is that you will only need to connect to IPv4 hosts on the internet. So it kind of defeats the purpose of setting up an IPv6 network other than to experiment with its features.

NAT64 protocol nats all outgoing IPv6 addresses to a pool of IPv4 addresses and then routes the request to the ISP. This means there will be double natting. Once for NAT64 and then again for your router to the ISP. To setup NAT64 install tayga.

"sudo apt-get install tagya"

edit /etc/tagya.conf

        tun-device nat64 # the interface to create to perform the natting
        ipv4-addr 192.168.255.1 # the ipv4 address for the tayg interface from the pool below.
        ipv6-addr fd5d:12c9:2201:1::3 # the IPv6 address to assign to the tun-device nat64 above
        prefix fd5d:12c9:2201:1:1:1::/96 # see explanation below
        dynamic-pool 192.168.255.0/24 # the pool of IPv4 addresses to assign to each IPv6 address
        data-dir /var/spool/tayga

So NAT64 creates a interface to listen on. The interface gets an IPv6 and IPv4 address. Any request from an IPv6 source for an IPv4 address will get routed to the nat64 interface. To do this we need to add a route for IPv6 addresses:

"sudo ip route add fd5d:12c9:2201:1:1:1::/96 dev nat64"

All that remains is to explain where the network prefix d5d:12c9:2201:1:1:1::/96 used above and in tagya.conf file come from. This is a network from your ULA block which you set aside for IPv4 to IPv6 address translation. You need to configure the bind DNS server to automatically convert any IPv4 addresses to IPv6 using the prefix above. This way you won't get an error when you try and ping an IPv4 host from an IPv6 address.

/etc/bind/named.conf.options file and add the items in bold below:

        options {
                directory "/var/cache/bind";
        
                 forwarders {
                        8.8.8.8;
                 };
        
                dns64 fd5d:12c9:2201:1:1:1::/96 {
                        clients {
                                any;
                        };
        
                        exclude {
                                any;
                        };
                };
        
                dnssec-validation auto;
        
                auth-nxdomain no;    # conform to RFC1035
                listen-on-v6 { any; };
        
        };

At this point your IPv6 network can live happily in a sea of public IPv4 addresses. Unfortunately you won't be able to access an IPv6 host with a global address just yet :(

Using Tunnel Brokers

Alternatively you can keep a dual stack and then setup a IPv6 tunnel at one of the well known tunnel brokers such as Hurricane Electric or setup a Teredo tunnel. Both have draw backs. The Hurricane Electric service requires you to setup a SIT tunnel to one of their points-of-presence. This requires a static IPv4 address or you will need to constantly update the configuration on their site when your ip changes. A Teredo tunnel requires a IPv6 address.

The advantages of a SIT tunnel is that you get a routable global IPv6 address at the POP and you can access IPv6 nodes natively. The disadvantage is your traffic needs to be proxied or tunneled over the IPv4 internet increasing latency. If you get a SIT tunnel you only have one global IP address for the host the tunnel is created on so you will need to setup a tunnel per node or have an IPv4 network and nat it through the SIT tunnel.

Risk of Using IPv6

One point to bear in mind when using IPv6 is that it will bypass all your IPv4 security measures. If you have an IPTables ruleset for IPv4 you will need to re-implement that for IPv6 with IP6tables. There is no natting in IP6tables currently. One thing to remember, if you in a co-location data centre and geting IPv6 auto-generated link-local addresses, any machine in the same centre can connect to your server via IPv6, potentially by passing any IPv4 ruleset you have setup! Some data-cenre might be assigning you global ipv6 addresses without you knowing! So check your co-location servers and make sure you don't have any back-doors due to mis-configuration. At least one data centre we have server at had helpfully given us an gobal IPv6.

Conclusion

The transition mechanisms are a mess. One just has to look at the acronym soup on Wikipedia transition mechanism page dealing with the myriad of ways to inter operate between IPv4 and IPv6. We have only tried two! We are still experimenting with the tunneling mechanisms above to see if they work properly as they promise the most benefits, being able to reach servers in different locations over the public internet with routable IPv6 addresses. So far we have met with limited success. No doubt we got to learn a bit more. But this also points to the biggest reason IPv6 is not being adopted. There is no real way for the two protocols to inter operate. A painful time will come when 50% of the world is on IPv6 and the rest on IPv4.

At least you can get some experience setting up internal IPv6 networks in the meantime.

Email Authentication Records: SPF, DKIM, and DMARC

2025-04-13T17:07:27Z

Sophisticated, at least from a social engineering point of view, syndicates operate in South Africa bombarding organisations with fake government tender or courier emails, often sending multiple scams daily. This two-part series explores how to protect your organisation from these, and other fraudulent emails.

In Part 1, we focus on email authentication protocols—SPF, DKIM, and DMARC—and reveal their strengths and weaknesses. Part 2 will cover non-authentication indicators of scams, such as suspicious links and behavioural patterns as well as some disturbing indication of compromised government servers or their users.

Cybersecurity Consulting

Expert services to protect your business.

Cybersecurity Training
CEH & DDevSecOps certifications and more.

SPF: Sender Policy Framework

SPF is a DNS TXT record listing authorised email servers for a domain. When an email arrives from user@example.com, the receiving server checks the SPF record against the sending server’s IP, using the MAIL FROM or Return-Path address—not the "From" address shown in email clients.

Senders can configure SPF to reject unauthorised emails or flag them as a soft fail. Without SPF, servers may trust the sender or use greylisting, increasing spam risks.

To check a domain’s SPF record, use:

dig -t TXT google.com

Look for a record starting with v=spf1, like:

v=spf1 include:_spf.google.com ~all

Note: SPF doesn’t verify the displayed "From" address, which spammers can spoof.

DKIM: DomainKeys Identified Mail

DKIM ensures email integrity using public/private key pairs. The sender’s public key resides in a DNS TXT record. The sending server signs headers (e.g., To, Subject, Date) and the email body, embedding a hash in the DKIM-Signature header.

The bh= tag confirms the body’s integrity. While the From header is often signed, DKIM doesn’t validate its authenticity.

Here’s an example DKIM signature from Google:

DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=google.com; s=20230601; t=1744370196; x=1744974996; darn=jumpingbean.co.za;
        h=to:from:subject:message-id:list-id:feedback-id:precedence
         :list-unsubscribe:reply-to:date:mime-version:from:to:cc:subject:date
         :message-id:reply-to;
        bh=UcP3oq8NmmBzZQi2XhAhYnWWRmOQ4WATXuFEpb6k+ww=;
        b=TxAXhYUdqgP0RFnLjMMPj9Hr8C2JRyKFrBypBtNqln+i/B3WRx+f/AGlUxxlNEuLNZ...

To retrieve the public key, query the domain (d=google.com) and selector (s=20230601):

dig 20230601._domainkey.google.com TXT

This returns:

"v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4zd3nfUoLHWFbfoPZzAb8bvjsFIIFsNypweLuPe4M+vAP1YxObFxRnpvLYz7Z+bORKLber5aGmgFF9iaufsH1z0..."

Note: DKIM allows spoofed "From" addresses to pass if the signed headers and body are intact.

DMARC: Domain-based Message Authentication

DMARC aligns SPF and DKIM results with the "From" domain. For example, an email claiming to be from example.com must have MAIL FROM (SPF) and DKIM domains matching example.com.

A strict DMARC policy (p=reject) blocks misaligned emails, even if SPF or DKIM pass for another domain. DMARC also provides reports to monitor authentication issues.

Check a DMARC policy with:

dig _dmarc.google.com TXT

Example response:

"v=DMARC1; p=reject; rua=mailto:mailauth-reports@google.com"

ARC: Authenticated Received Chain

ARC tracks SPF, DKIM, and DMARC results for emails passing through intermediaries like mailing lists, preserving their authenticity. We won’t cover ARC in detail here.

Limitations of Email Authentication

SPF, DKIM, and DMARC reduce email spoofing, but they rely on proper configuration. Missing DMARC records or misconfigured DNS entries allow spammers to fake the "From" address, bypassing SPF and DKIM unless DMARC enforces alignment.

Case Studies: South Africa Scam Emails

Examples use sanitized headers, but authentication details are accurate.

Spoofed Fastway.co.za Phishing Email


X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
Return-Path: <imodzeb@vps21150.dreamhostps.com>
Delivered-To: mark@recipient-domain.com
Received: from mailserver1.recipient-domain.com (localhost [127.0.0.1])
    by mailserver1.recipient-domain.com (Postfix) with ESMTP id CFED3B829E3
    for <mark@recipient-domain.com>; Sun, 13 Apr 2025 07:45:41 +0000 (UTC)
Received: from webserver1.sender-hosting.com [176.31.78.120]
    by mailserver1.recipient-domain.com with POP3 (fetchmail-6.4.38)
    for <mark@recipient-domain.com> (single-drop); Sun, 13 Apr 2025 07:45:41 +0000 (UTC)
Received: from vps21150.dreamhostps.com (vps21150.dreamhostps.com [69.163.197.224])
    by webserver1.sender-hosting.com (Postfix) with ESMTPS id EBE7A16E02CA
    for <mark@recipient-domain.com>; Sun, 13 Apr 2025 08:45:06 +0100 (BST)
Authentication-Results: webserver1.sender-hosting.com;
    dkim=none;
    spf=pass (webserver1.sender-hosting.com: domain of imodzeb@vps21150.dreamhostps.com designates 69.163.197.224 as permitted sender) smtp.mailfrom=imodzeb@vps21150.dreamhostps.com;
    dmarc=none
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed;
    d=signing-domain.com; s=mail; t=1744530307;
    h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
     to:to:cc:content-type:content-type;
    bh=kr/tXZjETnxLC9CK82Dp+VNXBQQgOR10CK28LSxzjNA=;
    b=tIWZANiYUPoYUsp96EsByFCxEtwQ5bP9Gm0aUe086x5OdDygpCaO/lT/v7UEyE6LfOCtsc
    6UMTlOxtn63xGyf5zo6BGx9keD9LzdvAgGtpCPwYgUjQ6R7E9Q5cOA0hbIQ4i7doJJED2J
    Wg3FinBm+0z2YerOn7/K9kOh/rAclzQ=
ARC-Authentication-Results: i=1;
    webserver1.sender-hosting.com;
    dkim=none;
    spf=pass (webserver1.sender-hosting.com: domain of imodzeb@vps21150.dreamhostps.com designates 69.163.197.224 as permitted sender) smtp.mailfrom=imodzeb@vps21150.dreamhostps.com;
    dmarc=none
ARC-Seal: i=1; s=mail; d=signing-domain.com; t=1744530307; a=rsa-sha256;
    cv=none;
    b=H2Q6K2fIhxLyjvHXi52wFP31veFojhXRvspFrKM/qWSCmhSL9Q1TfWQ7BXFtoRKPbUrIUr
    lU1PRxaugc744ocBPLzwWNhFN5MECENbuc2AQx88+MR2zhT+nzMzvDOfTYVeiGq4vtSPnS
    6yOk4xXo8RKxWLSMb8Vwquujwl7GxSc=
Received: by vps21150.dreamhostps.com (Postfix, from userid 6739095)
    id 4Zb1nS3r7FzRN7g8m; Sun, 13 Apr 2025 00:11:56 -0700 (PDT)
To: mark@recipient-domain.com
Subject: Your Shipment is on Hold
X-PHP-Originating-Script: 6739095:send.php
From: Fastway couriers <webmail@fastway.co.za>
Content-Type: text/html; charset=UTF-8
Message-Id: <4Zb1nS3r7FzRN7g8m@vps21150.dreamhostps.com>
Date: Sun, 13 Apr 2025 00:11:56 -0700 (PDT)
X-Spam-Status: Yes, score=11.66
X-Spam-Level: ***********
X-Spamd-Bar: +++++++++++
X-Spam: Yes

Analysis

This email claimed to be from

From: Fastway couriers <webmail@fastway.co.za>

It passed the SPF check against domain of imodzeb@vps21150.dreamhostps.com

Authentication-Results: webserver1.sender-hosting.com;
    dkim=none;
    spf=pass (webserver1.sender-hosting.com: domain of imodzeb@vps21150.dreamhostps.com designates 69.163.197.224 as permitted sender) smtp.mailfrom=imodzeb@vps21150.dreamhostps.com;
    dmarc=none

Why did the email server validate it against domain of imodzeb@vps21150.dreamhostps.com? Because of the Return-Path header:

Return-Path: <imodzeb@vps21150.dreamhostps.com>

No DKIM signature was presented by vps21150.dreamhostps.com so it could't be checked for integrity so maybe someone altered the email text along the way. Someone like another, more sophisticated spammer I suppose. Lastly since fasttway.co.za lacks a DMARC policy record it couldn't be used to detected the spoofed mail.

Dreamhostps SPF

We can check vps21150.dreamhostps.com SPF record with:

dig -t TXT vps21150.dreamhostps.com

This results in a record which tell us to check netblocks.dreamhost.com's SPF record:

vps21150.dreamhostps.com. 300 IN TXT "v=spf1 mx include:netblocks.dreamhost.com -all"

You can check netblocks.dreamhost.com records with:

dig -t TXT netblocks.dreamhostps.com

Fastways SPF & DMARC

We can check fastway.co.za SPF record with:

dig -t TXT fastway.co.za

This results in :

fastway.co.za. 300 IN TXT "v=spf1 mx ip4:103.61.69.0/24 ip4:101.0.80.178/32 ip4:101.0.80.179/32 ip4:101.0.80.180/32 ip4:101.0.80.181/32 ip4:192.168.32.165/24 include:spf.protection.outlook.com -all"

But as explained this was never looked up. We can check fastway.co.za doesn't have a DMARC record with:

dig -t TXT _dmarc.fastway.co.za

This results in an empty response. So DMARC is not configured at the time of writing.

Not Mail Authentication Indicator of Phishing

The email body included a suspicious link (https://swingzbegonacervera.com/test), further indicating spam. I might do another article on this at some future date. The syndicates operating in South Africa has a particular set of techniques and tactics that they follow that can be used to identify phishing emails.

Cybersquatting: dsd-govtenders.online

-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
Return-Path: <zikhona.sodika@dsd-govtenders.online>
Delivered-To: mark@sanitised-domain-one.com
Received: from sanitised-domain-two.com (localhost [127.0.0.1])
    by sanitised-domain-two.com (Postfix) with ESMTP id 2562EB82933
    for <mark@sanitised-domain-one.com>; Fri, 11 Apr 2025 14:25:14 +0000 (UTC)
Received: from sanitised-domain-three.com [176.31.78.120]
    by sanitised-domain-two.com with POP3 (fetchmail-6.4.38)
    for <mark@sanitised-domain-one.com> (single-drop); Fri, 11 Apr 2025 14:25:14 +0000 (UTC)
Received: from JN3P275CU003.outbound.protection.outlook.com (mail-southafricanorthazon11021136.outbound.protection.outlook.com [40.107.141.136])
    by sanitised-domain-three.com (Postfix) with ESMTPS id DFEE016E00AD
    for <mark@sanitised-domain-one.com>; Fri, 11 Apr 2025 15:24:18 +0100 (BST)
Authentication-Results: sanitised-domain-three.com;
    dkim=none;
    arc=pass ("microsoft.com:s=arcselector10001:i=1");
    dmarc=none;
    spf=pass (sanitised-domain-three.com: domain of zikhona.sodika@dsd-govtenders.online designates 40.107.141.136 as permitted sender) smtp.mailfrom=zikhona.sodika@dsd-govtenders.online
ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed;
    d=sanitised-domain-one.com; s=mail; t=1744381460;
    h=from:from:reply-to:subject:subject:date:date:message-id:message-id:to:
     cc:mime-version:mime-version:content-type:content-type;
    bh=yoYEgrf3l9z7INabfXJ2vAybE/abtR7hsVhzvUAXyhU=;
    b=sPHUeukaI4QDAbQj7SrrEwXy9KZqOEA7oX62U+R8DbHY2oK13zb4UgRsNXV7V0PpVtL/ZR
    dzpyMoqmZZcp3AwATLLQPnn8VjIfFqhb0x4Geb5u8/zdoQjsPx1euvL4DKJh0GBN7VkKDW
    FYqe397t2QJg4vnc+Lpk4FlkJ4G1ue4=
ARC-Authentication-Results: i=2;
    sanitised-domain-three.com;
    dkim=none;
    arc=pass ("microsoft.com:s=arcselector10001:i=1");
    dmarc=none;
    spf=pass (sanitised-domain-three.com: domain of zikhona.sodika@dsd-govtenders.online designates 40.107.141.136 as permitted sender) smtp.mailfrom=zikhona.sodika@dsd-govtenders.online
ARC-Seal: i=2; s=mail; d=sanitised-domain-one.com; t=1744381460; a=rsa-sha256;
    cv=pass;
    b=dPxdGRXIPqX9DRI9BtVDjm8Ls786WtX/jpIVw2w1diHOJqkDRhx8L/C19SN1VOvF4UQ75p
    wnPo3TKJO0Vb7RaxCil9K9ATA1vvck9jcfD0ZtRIX9vLu1SIjcvdjz1RYsVCGBFUjXY9j+
    C9SOi8u9iH7OSjQKVtEPH3S0jiaKMos=
ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none;
 b=HyyhvFKT/a2UUfqs+55LVKYjvoKN7+9RGfT45R3SJwHb0PMCVjw0WyuOhTDrxpCSV1vNJV2b9/jW0xAA5lsy1PUlJdL9Rmv4sQNlCgRRKQNuGSTmfLGV9VPdY3WWGTZFD9bt3wEWmhEdQhy6lou4Q3POUsbOxyGrdeFB/iI5D8DeA8acE7Tnhx/St4df75nT8pbUv5nnY4GAyEwPCoyk/DtD77frDlLEzsP63FJpvzEvSwvXU1OAaJRY4D7AuA8H99SM2RVRtx77q5rUUumVp7aTFi2v82afkzTcTNMfdTe6HBjoEoYpWyF6oECEiiCvRSMH/mvFKH6V2opBqcwwMQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector10001;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=yoYEgrf3l9z7INabfXJ2vAybE/abtR7hsVhzvUAXyhU=;
 b=MTZfy3Ru/T0IUPJjyIJhKg/kDWn/5e4DZkIksEGOuNXAMlPsClgA8kKnsEUc6NvJba3++MhPcxzVjvT56PgX1xtFz83PavCMk8e78WHKgEdTNYmHj3M6r6sZFVWXoHOF+qSD84ouVCGSrAQqMVDDR7n8ANwXj1SzS7TaUrnI5XBMpBukxeNzpop1x/1EFdxc2sk08WPEqgwVkiTwgFuXEWrSbQYB81J+3Po2vBOq3mw7T8pPJPoNjZu54I/rMKl/OuTTVagU/OKrB1kJXiGlcH5BGMQ5WNhK3LrM55c783+DhqzGxjMdjAqsw5MVciWgQrt4FI5PQiDm150pkl6Kvw==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=dsd-govtenders.online; dmarc=pass action=none
 header.from=dsd-govtenders.online; dkim=pass header.d=dsd-govtenders.online;
 arc=none
Received: from CP7P275MB1559.ZAFP275.PROD.OUTLOOK.COM (2603:1086:100:3e::11)
 by JN3P275MB2712.ZAFP275.PROD.OUTLOOK.COM (2603:1086:0:bb::5) with Microsoft
 SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
 15.20.8632.27; Fri, 11 Apr 2025 14:08:57 +0000
Received: from CP7P275MB1559.ZAFP275.PROD.OUTLOOK.COM
 ([fe80::2910:b44a:8e71:32d3]) by CP7P275MB1559.ZAFP275.PROD.OUTLOOK.COM
 ([fe80::2910:b44a:8e71:32d3%7]) with mapi id 15.20.8632.025; Fri, 11 Apr 2025
 14:08:57 +0000
From: zikhona sodika <zikhona.sodika@dsd-govtenders.online>
Subject: Seeking Immediate Service Provider
Thread-Topic: Seeking Immediate Service Provider
Thread-Index: AQHbquq6V9sN+KEKtEiv7XNhKuBfLQ==
Date: Fri, 11 Apr 2025 14:08:57 +0000
Message-ID:
 <CP7P275MB1559C0F9FD8A280351263C30C1B62@CP7P275MB1559.ZAFP275.PROD.OUTLOOK.COM>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator:
msip_labels:
x-ms-publictraffictype: Email
x-ms-traffictypediagnostic: CP7P275MB1559:EE_|JN3P275MB2712:EE_
x-ms-office365-filtering-correlation-id: 8222b1d6-6690-4ae6-b6e6-08dd790266a1
x-ms-exchange-senderadcheck: 1
x-ms-exchange-antispam-relay: 0
x-microsoft-antispam:
 BCL:0;ARA:13230040|7416014|376014|1800799024|366016|7055299006|10085299006|8096899003|4053099003|38070700018|27013499003;
x-microsoft-antispam-message-info:
 =?iso-8859-1?Q?AxjxpuYCDzACm2gNeLj/BnAgl1OM/ytXBKbgUBdnMhwUYfA26pWOjj8wDr?=
 =?iso-8859-1?Q?TJxAj5xC3SRATsgMHIgk793p/4gXaylW6WUNP7yJWoBL3rTitRj7wW55k9?=
 =?iso-8859-1?Q?QUo1Za6O2VRX4vR/xG+vCSFCyhb9v9xyCqqR6Q3kT5WRBRUhkUCi5ebap7?=
 =?iso-8859-1?Q?YQ1hji8zSk38HVYpT9uBAz3EYaw449CzJhO4krqdDdM6NPvCHLDnyzqu2g?=
 =?iso-8859-1?Q?wcF9sCC/mkXa2CHUlYq2Dp3x2YC3l/e4xgwLrb7uOXIk+DLG/iZCweT9yn?=
 =?iso-8859-1?Q?VHsxyvFaNdeQhkEZ33tAjQe3074EFrHJy8KWN62LhLTYKeg+Gbmn0sDAE8?=
 =?iso-8859-1?Q?lg39svLPGn9WtXP8iG3alSqo75EB7jvytLcC7+W/+V9lZYbi+8QtAj8jiG?=
 ...(edited)
x-forefront-antispam-report:
 CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CP7P275MB1559.ZAFP275.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(7416014)(376014)(1800799024)(366016)(7055299006)(10085299006)(8096899003)(4053099003)(38070700018)(27013499003);DIR:OUT;SFP:1102;
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0:
 =?iso-8859-1?Q?0RR4dLnFtVa6l9QD5Tpv9ekvnvy2FTzNyR8akfH+be3QnwSCGUEZ4eXfXi?=
 =?iso-8859-1?Q?aZJJObThtTk9Pt+CmijcRVBd3iMorsQt5X+zsptyfH7Ph4KcvPY85ZSH4l?=
 ...(edited)
Content-Type: multipart/mixed;
    boundary="_004_CP7P275MB1559C0F9FD8A280351263C30C1B62CP7P275MB1559ZAFP_"
MIME-Version: 1.0
X-OriginatorOrg: dsd-govtenders.online
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: CP7P275MB1559.ZAFP275.PROD.OUTLOOK.COM
X-MS-Exchange-CrossTenant-Network-Message-Id: 8222b1d6-6690-4ae6-b6e6-08dd790266a1
X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Apr 2025 14:08:57.5088
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 9a580612-8441-4101-9b38-0037105854c3
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: l0CMIWIqn/mxzNqrfAaeHucbnB7k8AqY0/JD7yiFFYIT1pVEc6OhvUUaXp7qBgmJIf2OeOiilUfDDCgFofU93fo27V14SGx0G24aWdTzbj23HU/6uZVaCoSit8GrwHGa
X-MS-Exchange-Transport-CrossTenantHeadersStamped: JN3P275MB2712
X-Spam-Status: Yes, score=8.10
X-Spamd-Bar: ++++++++
X-Spam-Level: ********
X-Spam: Yes

--_004_CP7P275MB1559C0F9FD8A280351263C30C1B62CP7P275MB1559ZAFP_
Content-Type: multipart/alternative;
    boundary="_000_CP7P275MB1559C0F9FD8A280351263C30C1B62CP7P275MB1559ZAFP_"

--_000_CP7P275MB1559C0F9FD8A280351

This email used a "fake" domain, dsd-govtenders.online with request to quote on a fake tender hoping that the recipient doesn't look to closely at the sending email domain. The domain dsd-govtenders.online is a "legitimate" domain owned by the scammer. Since they own the domain they can generate SPF, DMARC and DKIM records for the domain. They hope that the SPF passing is enough to convince you it is legitimate.

Despite this the Authentication-Results show that there is no dkim nor dmarc set up:

Authentication-Results: sanitised-domain-three.com;
    dkim=none;
    arc=pass ("microsoft.com:s=arcselector10001:i=1");
    dmarc=none;
    spf=pass (sanitised-domain-three.com: domain of zikhona.sodika@dsd-govtenders.online designates 40.107.141.136 as permitted

This is only slightly better than some who try to set it up DKIM and DMARC but get it wrong. The syndicates like using godaddy hosting which is a common indicator of these tender scam emails.

The use of exchange by godaddy may also give the email a false sense of legitimacy. I mean Exchange's anti-spam test passes with flying colours right? The fact that one needs a credit card and other identification to register with godadddy makes it even more perplexing why law-enforcement can't seem to stop these criminals.

We can checking the SPF records for dsd-govtenders.online :

dig dsd-govtenders.online TXT

We get the response:

dsd-govtenders.online. 3600 IN TXT "v=spf1 include:secureserver.net -all"

All the servers are owned by godaddy. There is no DKIM header and there is no DMARC:

dig _dmarc.dsd-govtenders.online TXT

No DNS records are returned. Looking at when the domain was resisted with:

whois dsd-govtenders.online

We can extract the following date:
Creation Date: 2025-02-06T17:12:24.0Z

Hmmm doesn't look legite.

Conclusion

SPF, DKIM, and DMARC are critical for combating email spoofing, but they’re only effective when properly configured. South Africa’s tender scam syndicate exploits missing DMARC records and lax configurations to deceive organisations. By analysing email headers and DNS records, you can identify these scams. Stay tuned for Part 2, where we’ll explore additional red flags like suspicious links and social engineering tactics.

Application Password Best Practice

2025-04-13T08:40:04Z

Submitted by Mark Clarke on Wed, 06/10/2015 - 08:57

Note: This article is old and needs to be updated but is reproduced here for history. It doesn't cover API tokens, oAuth2/OpenID, Kerberos etc.

Our Certified Ethical Hacker CEH training course is attended by all types, from system administrators and application developers to cyber security professionals. A common question from developers is how to implement secure passwords in applications when there is no LDAP, Kerberos or Active Directory integration.

This question comes up in the sections of the CEH training covering password cracking techniques, password algorithms and best practice so its appropriate to cover what these are first.

Certified Ethical Hacker (CEH) Training
&
Certified Security Analyst Training (ECSA)

Password History

Back-in-the-day it was thought it was enough to merely hash user passwords in a one-way encryption algorithm such as MD5. If the password was revealed, for example via someone getting read access to the password database or via an unencrypted network connection, assuming the password is being transferred encrypted, the perpetrator would not have access to the password.

A lot of protocols and application still use password authentication protocol (PAP) where plain text passwords are passed around until they reach the password database where they are encrypted and then compared to the stored encrypted password for authorisation.

In early versions of Unix, for example, encrypted password were stored in the world-readable /etc/passwd file until they were moved out to the shadow password file readable only by root. It was soon realised that this was not a good idea. The first step is never to let anyone get hold fo the encrypted password :)

Password Encryption

The principle is, given a sound encryption algorithm, the amount of computing time required to crack an encrypted password using that algorithm should makes it infeasible or uneconomic to perform. But, as computing power increases, so the algorithms used to encrypt need to be enhanced or changed to keep the cost curve high enough. Additionally some formally "secure" algorithms are found to have fatal flaws some time after adoption.

MD5, for example, is now considered insecure due to a flaw in the algorithm and as a way has been found to generate collisions, ie the same hash is generated for different input.

SHA1 algorithms are still considered secure but as computing power is increasing it is consider a matter of time (years) before computers are commonly available that can crack a SHA1 password. Everyone is upgrading to SHA2.

"Uncrackable" Encryption Algorithm

So, assuming you have an "uncrackable" algorithm, is it enough for password security? The answer is, of course, no. If someone has an encrypted password they can simply attempt to brute force it. i.e guess the password by trying different combination of characters until they get the matching hash, hence those minimum password requirements.

Even if proper password policies are enforced such as minimum length of 10 characters or more with a combination of upper case and lower case etc password as still vulnerable simply because people will tend to use common combinations of letters and characters, thereby limiting the key space to a size that becomes feasible to crack.

Rainbow Tables

Still brute forcing passwords with a good policy is time consuming. To speed up attacks hash password can be pre-generated and then compared against any encrypted password via lookup to check for a match. This speeds up password "cracking" by orders of magnitude as one no longer needs to laboriously encrypt every combination in the key space and then compare it to the actual password aka brute forcing.

These pre-generated tables of passwords are commonly called rainbow tables and are one of the first things run against a password database once obtain.

Password Salting and Hashing

So even if you force your users to use strange combinations of keys, users will probably use the common special characters of "!","#" and digits and common passwords across applications and services. If I get your password from a service that is less secure I can then simply compare encrypt it with a different algorithm and compare it to another database for a match.

An technique to work around rainbow tables, reused or common passwords, that heightens the cost of cracking is to use what is referred to as a salt. A salt is a random combination of characters used as a prefix or suffix to a user password which is then encrypted and stored. The salt also needs to be stored along with the password otherwise it would not be possible for the algorithm to generated the encrypted password again.

The benefits of this is that rainbow tables cannot be used, forcing the person trying to crack the password back to brute-forcing it. The important point to note is that the salt is kep along with the password so if the database is stolen the perpetrator has access to the salt as well. So they can still brute force passwords but not use rainbow tables. (Unless of course you slat generator has a limited key space too and then they just try and pre-generate all combinations thereof.)

Application Password Best Practice

So to implement a good password scheme for your application you need:

A good algorithm, preferably one which can increase cost of computation over time without requiring algorithm changes
A good salting generator and a
A protocol to store the salt and encrypted password

Bcrypt Algorithm and Password API

The state-of-the-art encryption algorithm for password encryption, that is also easy to use, is the Bcrypt algorithm.

Bcrypt is based on the Blowfish cipher, uses salts to encrypt passwords and is an adaptive algorithm. The algorithm incorporate an iteration count, the number of times the password is hashed, that can be increased over time. This ensures that the cost to compute the encrypted password increases even as computing power increases simply by increasing the number of iterations.

As an application developer you will not have to change any code to generate stronger encryption, simply read the number of iterations from a configuration file to generate or regenerate an encrypted password and you good to go. Of course this assume there is no flaw discovered in Blowfish at some point.

The are many libraries out there for bcrypt for your favorite language. We use the PHP and Java libraries and using it couldn't be simpler. Below is an example in Java.

import org.mindrot.jbcrypt.BCrypt;

public class UserService {

          public boolean savePassword(User user,String password){
                ...                   
                String enc = BCrypt.hashpw(password,BCrypt.gensalt(20));
                ...  
          }

          public boolean authenticate(String username,String password){
                ...
                String enc = getUserPassword(username);
                if (enc!=null){
                        return BCrypt.checkpw(password,enc));
                } else{
                        return false;
                }
          }
}

Just remember to make it hard to get to the user database in the first place.

Process Substitution - Linux Expert Tip

2025-04-13T08:19:05Z

So I am sure you have all heard of, and used, command substitution and parameter substitution or expansion but process substitution?

LPI Linux Training By Professionals

Process substitution allows you to send stdout of one process to stdin of another process. Bah you say! This is just piping stdout to stdin and can be done simply with pipes (|). What is so awesome about chaining together standard streams?

Yes, it is true that pipes can be used to pump the stdout of one process to stdin of another and is trivial to implement, but what if you have a command that takes two inputs not just one? Or if you need to branch of processing into two different pipelines i,e duplicate the intermediate results on stdout and process them differently from then on?

Since you can only redirect one stream's stdout or stdin piping won't help. What happens if the command reads from or writes to a file only and not stdin/stdout? In this case you are left with creating temporary files to hold the intermediate output for consumption by subsequent commands. But what is I told you there is an easier way?

Process substitution allows the stdout of one command to appear as a file for consumption to subsequent commands, or allows a command that expects to write to a file to write to the stdin of a subsequent command instead.

Using files to store intermediate results

Here is an example if we do not use process substitution. The diff command takes two files as input and outputs the difference between the two files. Lets say we wanted to get the difference between two directories. We would need to do something like this:

ls -l /home/tux > file1 ls -l /home/tuxbk > file2 diff -u file1 file2

We couldn’t just pipe the commands together like so:

ls -l /home/tux | ls /home/tuxbk | diff -u

But we can do this all in one line with process substitution

Process Substitution Syntax

First lets look at the general process substitution syntax. The general syntax for process substitution is <(command) for reading from a command's stdout instead of a file and >(commands) for writing to a processes stdin instead of a file. Note: there is no space between the angle bracket and the left brace! Let first looks at a simple, and useless, example:

We can run wc as follows:

ls -l | wc

Outputs:

168 1512 13430

The above can be done as follow using process substitution:

wc <(ls -l)

Outputs

168 1512 13430 /dev/fd/63

The results are the same but notice the file descriptor handle (/dev/fs/63) that is displayed in the output for the substitution syntax. You can think of it as process substitution creating a temporary file to handle the intermediate results for consumption by a later command This (/dev/fd63) is the intermediate file created to send the results between two processes.

Process Substitution - Reading from a process's stdout instead of a file

So now lets looks at something more interesting. I introduced the diff command above to display the changes between two directories. with process substitution we can now do this in one line:

diff -u <(ls -l /home/tux) <(ls -l /home/tuxbk)

Above we have replaced the two file descriptors, which the diff command will read from, with the output of two processes.

Some Awesome Process Substitution Examples

The power of process substitution can be further demonstrated with the join command. The join command takes two sorted files as input and joins matching lines on the first to lines in the 2nd file, discarding lines which do not match.

Armed with this we could join the output of the top command to the output of the iotop command. Note if you are using sudo with a password you will need to prime the cache with the users password to avoid an error when being prompted on the first run. You will also have to install iotop if you don't have it already.

join <(top -b -n 1 | sed '1,7d'| sort -n) <(sudo iotop -P -b -n 1 | sed '1,3d' | sort -n)

The above command matches pid from top with pids from iotop and combines the output. There are simpler ways of doing this with sar but it does illustrate the point quite nicely.

Splitting your process pipeline with Tee and Process Substitution

And the best place to use process substitution is with the tee command. Traditionally we use the tee command to pipe intermediate output to a file and then continue to process the output down our pipeline but with process substitution we can do even better!

Process Substitution - Writing to process's stdout instead of a file

With tee and process substitution we can split output from a series of commands into two parallel streams of execution. First a simple example of the tee command.

grep -i "error" /var/log/syslog | tee errors.txt | grep apache > apache.error

The above simply greps for the pattern "error" and then saves all lines to errors.txt and then we extract just those error lines that relate to apache. But what if we really wanted to extract errors for mysql into a separate file as well as the apache errors? We can use process substitution:

grep -i "error" /var/log/syslog | tee >(grep mysql >mysql.error) | grep apache > apache.error

Another example to calculate MD5Sums and SHA256Sums of files in a directory

find ./ -type f | tee >(xargs -n1 md5sum >md5sums.txt)| xargs -n1 sha256sum >sha256sums.txt

You will notice that we have redirected stdout in the tee command to a file. If you didn't do this then the stdout of the process substitution commands would be piped to the subsequent commands too - which mean you will have a fork and join kind of pipeline!

Process Substitution - The Best Kept Shell Secret!

Isn't that awesome? Now go forth and do magic in the world!

Ghost in the subshell

2025-04-13T08:01:33Z

Ghost in the subshell

Submitted by Mark Clarke on Thu, 08/03/2017 - 14:27

Bash is a deceptively simply interpreter, seemingly easy to use but harder to master its subtleties. But this is what makes free software so much more fun and intellectually stimulating than stuff rushed out a corporate door with deadlines and minimal viable product mentality.

Bash Pipelines and Subshells

One of those shell subtleties that takes a while to grok is subshells and in this case subshells invoke via pipes. Most people are familiar with bash pipes and redirection of "stdin" and "stdout". If not you should attend one of our Linux training courses :).

With pipes we can architect solutions from various components (commands & shell functionalities etc) to perform a task impossible to achieve with a single tool alone, and for which no specific utility exists. One subtlety about pipes is that, besides redirecting input and output streams, the shell invoke a subshell for each command in the pipeline. i.e child processes that run in parallel with each other rather than serially are created. Lets look at a trivial example using pipes.

When we see a pipeline such as

cat /etc/passwd | tr a-z A-Z

instinctively we assume the "cat" command is run, then its output is passed to "tr" and then the "tr" command is run. Although it might appear to us that the commands are running serially this is a misconception. In fact both commands are running concurrently, it is just that the command on the right is waiting on its standard input stream for data, which it is getting from the command on the left. To prove this we can use the a pipeline with two where the right command is not dependent on the left commands output.

To demo this we will also introduce the "time" commands. As it's name suggests this provide timing information on commands. e.g.

time sleep 2

should show an elapsed time slightly longer than 2 seconds. So if we took two sleep commands and piped them together like so

time sleep 2 | sleep 2

What time would we expect to see? If they ran serially then it would be slightly more than 4 seconds
but it they run in parallel the time would be slightly more than 2 seconds. As you see if you run the command they time is slightly more than 2 seconds - showing the commands are running in parallel.

Improve Bash Performance with Subshells

So how can we use this functionality? Lets say you have a long list of numbers you need to sum, how would you do this in bash and how can we optimise it?

First some explanations on our examples to follow so that we can separate out new commands from the subshell concept we are trying to show . To simulate a "list of numbers" we will generate a sequence from 1 to 9,000,000 using the "seq" command, and then add them sum them all up. ("seq 1 9000000")

Yes, that's right basically we are going to calculate the sum of a range of integers. We are using a "large" range from 1 to 9 million so we can actually see the difference when we optimise our initial approach later. For a processor doing billions of operations a second this workload is no challenge. On faster machines you may need to up the range to be able t observe any improvements.

We could, of course, do this calculation in constant time O(c) using Gauss's formula but that would reduce the didactic value of this post :) So to add up the range generated by "seq" we will use "awk" and the "time" command to get some readings for comparison.

Awk, Seq and Some Output Redirection

Note because we are using a sequence of commands and passing them to the "time" command we need to let "time" know we want to measure the execution time of the entire pipeline not just the first command so we have to wrap our command up in "sh -c" call. We also have to escape the "awk" script so bash doesn't try and interpret the $ variables as bash variables. Later when we put this in a script it will be much better.

time bash -c "seq 1 9000000 | awk 'BEGIN { SUM=0 } { SUM=SUM+\$0 } END{ printf \"%13.0f\", SUM }'"

Running this 4 times on my desktop gave me an average time of 0.7355s. We can check the accuracy of the calculation with Gauss's formula -

$(( 9000000(9000000+1)/2 ))

So how can we speed this up? As already stated pipelines do more than "funnel stdout to stdin", they also invoke each command in a subshells or child process of the parent process. Essentially both commands are running in parallel as we explained above. The entire pipeline exits once all subshells processes return. So if there are no dependencies then the longest running process in the pipeline determines the run time of the pipeline.

Subshells, Scatter Gather and Named Pipes

We can use the fact that each command is running in a separate process to split up our number ranges into several concurrent processes. have them summed individual subsets and then gather the results and aggregate them in a map/reduce scatter/gather type of algorithm.

We are going to achieve this by dividing our sequence into 3 subsets and launching explicit subshells for each data set and sum the subsets up. We will use a pipe to gather the results of the subshells and aggregate the results. (If this was a file containing a long list of numbers we could use "sed -n p1,x numbers.txt" to apportion the file.)

First lets create a pipe with the "mkfifo" command. A named pipe is just like a bash pipe except its lifetime is independent of any command that us it, it can be written to by more than one command at a time and read from by more than one command at a time. We are going to use this ability to be written to by more than one command to gather the results of our separate subshells.

mkfifo ~/pipe

With our pipe established we can use it in our process as follows:

time bash -c "seq 1 3000000 | awk 'BEGIN { SUM=0 } { SUM=SUM+\$0 } END { printf \"%13.0f\\n\", SUM }' > ~/pipe | seq 3000001 6000000 | awk 'BEGIN { SUM=0 } { SUM=SUM+\$0 } END { printf \"%13.0f\\n\", SUM }' > ~/pipe | seq 6000001 9000000 | awk 'BEGIN { SUM=0 } { SUM=SUM+\$0 } END { printf \"%13.0f\\n\", SUM }' > ~/pipe | awk 'BEGIN {SUM=0} {SUM=SUM+\$0} END {printf \"%13.0f\" ,SUM }' <~/pipe"

Sjo, that's a lot of text so lets break it down. The first 3 core command sequences are just the same as our initial command except the results are redirected to the named pipe. Each "step" handles a subset of the range 1 to 9,000,000.

seq 1 3000000 | awk 'BEGIN { SUM=0 } { SUM=SUM+\$0 } END { printf \"%13.0f\\n\", SUM }' > ~/pipe

Since the stdout of each one fo these steps is written to a pipe they all run independently. In fact each "step" invokes two "linked" subshells to get its job done.

The last step is aggregating the totals from the 3 steps by reading from the pipe

awk 'BEGIN {SUM=0} {SUM=SUM+\$0} END {printf \"%13.0f\" ,SUM }' <~/pipe

Improved Performance with Subshells

On my system the average time for 4 iterations was 0.27325s. This is a massive improvement over the 0.7355s it took to complete the task in a single subshell. Its more than 2.5 times faster.

To make this easier to read and less repetitive we can put the subset sum step into a function and invoke it in a script.

#!/bin/bash set -e set -o pipefail function sum { seq $1 $2 | awk 'BEGIN { SUM=0} {SUM=SUM+$0 } END { printf "%13.0f\n", SUM }'; } sum 1 3000000 > ~/pipe | sum 3000001 6000000 > ~/pipe | sum 6000001 9000000 > ~/pipe | awk 'BEGIN {SUM=0} {SUM=SUM+$0} END {printf "%13.0f\n" ,SUM }' <~/pipe exit 0;

Understanding subshells in Bash is one concept which takes time to really grok. Its quick to pick up the pipe character as a way to redirect stdin and stdout but it is so much more subtle than just that. There are commands such as gnu parallels which take parallel processing to a whole new level.

The Future of Tech:Fast-Growing Career Options

2024-02-27T10:51:11Z

Careers in Information Technology (IT) offer unparalleled opportunities for growth, innovation, and impact. From cybersecurity to cloud computing, software development to data analysis, the realm of IT encompasses a vast array of specialized roles that play crucial roles in driving the digital transformation of businesses and organizations worldwide. As technology continues to permeate every aspect of our lives, the demand for skilled IT professionals is skyrocketing, making it an exciting and lucrative field for aspiring individuals.

However, with such diversity and dynamism comes the challenge of navigating the multitude of career paths, certifications, and specializations available. In this comprehensive guide, we delve into the multifaceted world of IT careers, exploring the myriad opportunities, challenges, and strategies for success in this ever-evolving industry.

Whether you're a seasoned professional looking to advance your career or a newcomer seeking to break into the field, this article aims to provide invaluable insights and guidance to help you thrive in the dynamic realm of IT. We are here to assist with your IT career. Contact Us.

AI and Machine Learning Specialist: With AI and machine learning becoming integral to various industries, specialists in this area are in high demand. Careers focus on developing algorithms that can learn from and make predictions on data.
Quantum Computing Scientist: As quantum computing moves from research to reality, professionals with expertise in quantum mechanics and computer science are needed to develop new algorithms and applications.
Cybersecurity: With the increasing frequency and sophistication of cyber-attacks, cybersecurity experts are crucial for protecting sensitive information across all sectors, from government to healthcare.
Data Scientist and Analysts: The ability to analyze and derive insights from big data remains a highly valued skill, with applications in every sector from finance to healthcare to marketing.
Blockchain Developer: Beyond cryptocurrencies, blockchain technology offers applications in secure and transparent transaction processes, supply chain management, and more, leading to increased demand for skilled developers.
Cloud Solutions Architect: As businesses continue to migrate to cloud services for scalability, flexibility, and efficiency, architects who can design and manage cloud computing strategies are essential.
IoT Solutions Specialist: With the proliferation of connected devices, specialists who can develop and manage IoT solutions are needed to harness the power of this technology for smart homes, cities, and industries.
Ethical Hacker or Penetration Tester: Professionals who can ethically breach systems to identify security weaknesses and vulnerabilities are critical in the fight against cybercrime.
Augmented Reality (AR) and Virtual Reality (VR) Developer: With applications in entertainment, education, and training, AR and VR developers are in demand to create immersive experiences.
Sustainability in Tech Expert: As the focus on environmental sustainability grows, professionals who can integrate green practices into technology design, development, and deployment are becoming increasingly important.
Digital Transformation Consultant: Experts who can guide businesses through digital transformation, helping them to implement new technologies and digital practices, are crucial for companies looking to stay competitive.
Privacy Officer/Consultant: With increasing regulations around data privacy (e.g., GDPR, CCPA), professionals who can navigate the legal and technical aspects of data privacy are essential.
Edge Computing Specialist: As edge computing grows to process data closer to where it's generated for improved speed and efficiency, specialists in this area are needed.

Managing the Risk of OpenJDK Migration

2023-09-06T09:19:41Z

Life as a Java developer used to be about writing awesome code. Understanding the latest APIs and hottest trends in application design and architecture and how you could leverage this all for fun and profit. For those that wanted to, you could delve into the fascinating or hellish depth of the Java VM, garbage collection and performance tweaking. One looked forward to a new version of Java and all the goodness it brought. It was not like those other runtimes where every release meant work to update your code base just to get what you already had. If you had the inclination you could recompile to get some performance boosts and it didn't require too much bureaucratic nonsense to upgrade from one version to the next and which distribution to use. DevOps did away with those pesky sysops roadblocks too about staying on older versions. In 2017 life got even better we had a new release cycle that promised even more goodness, more often. Times were good.

JDK Licensing - A new headache

Then one day it changed, slowly at first. In 2019 Oracle announced it would end free public updates for commercial use of Java 8 implementation of Java and if you installed any updates you would be liable. The writing was on the wall and the future was looking a little darker. There were some more gyrations in 2019 that confirmed the cloudy outlook for Oracle JDK and alternative OpenJDK distributions looked better and better. Then came Covid and lockdowns and we had other things to worry about. We emerged from that dystopian aberration to be hit with more licensing changes. Even to those who hoped that the storm would pass them by, it was now clear one needed to invest more time and energy into the choice of which JDK distribution to use. Keeping up with the constant changes is tiring, not to mention unproductive and looked set to continue. Unless you enjoy this type of thing, In the words of Sweet Brown "Ain't no one got time for that". I once met someone who proudly told me he had passed a vendor licensing exam. Thats right. The licensing was so complex one needed to be certified to be able to sell it. The last thing one needs is audit and compliance knocking at the door asking questions with spreadsheets and PowerPoint presentations.

Looking for help to manage your JDK Challenges?

Thankfully Simmon Ritter's new book "OpenJDK Migrations for Dummies" is here to help you understand the issues around the new world of Java VM licensing, how to choose an OpenJDK Distribution provider and plan and execute a migration strategy. As a developer, I enjoyed the chapters on "Preparing for Your Migration" and "Migrating Your Applications" the most as they deal with the technical requirements. It's really nice to have a place where all the potential issues with JDK versions are documented in a clear and concise manner. A reference guide so to speak. Appendix B "Optimising the JVM for Lower Latency, Higher Throughput and Faster Warm-up" and Appendix C "Runtime Security" were great too.

Manage your OpenJDK financial and compliance risk

The remaining chapters deal with the tedious but vitally important task of managing your Java JDK usage. Enterprises need to seriously consider the risk they expose themselves to if they don't undertake an exercise to understand their JDK usage both financially from a licensing point of view and from a cybersecurity and resilience perspective. A team needs to be assembled and a project undertaking to address the potential risks and plot a way forward. The book provides handy checklists and schedules to audit and record your Java usage and how to plan your migration as well as migration strategies. it provides guidance on choosing an OpenJDK Distribution Provider covering issues such as supported versions, release cycles and types and levels of support as well as the all-important issues around cost and compliance. If you are a Java Developer and in management and you need to undertake the unenviable task of assessing your Java environment Simmon Ritter's book is an invaluable guide to making the right choice for your business.

Upgrading to Flectra 2 - A Journal

2024-03-22T13:35:30Z

Upgrade to Flectra 2 - A Journal

A few years ago, just before Covid, we were looking for an e-commerce engine to start our online shop that was spun out of our main business. We were also looking for an accounting system to replace the Windows-based system that our succession of bookkeepers insisted on using. When our last bookkeeper emigrated we felt it was an opportune time to bring it all in-house and rely on an open-source system where we could take it places we never considered before with a closed-sourced system and, at the same time, begin to offer support and subscriptions from the creators to our customers who would find the product as useful as we did.

We looked at open-source accounting systems about a decade ago, sans the e-commerce requirement. At the time the main contenders were OpenBravo, Tiny ERP and SQLedger. We didn't settle on any of them at the time. Our risk assessment was as follows:

OpenBravo - too complex, the project did not have a track record yet so it's future was uncertain. The community was still nascent.
SQLedger - Written in Perl, looked like development had stalled and was butt ugly but could meet our limited accounting requirements at the time.
Tiny ERP - too complex for our requirements at the time and was client based.

None of these projects would have turned out to be a good option to bet the company accounting system on. OpenBravo, stopped its open-source editions in 2019. Seems they regret the use of Open in their name now. Tiny ERP became Odoo and, as I will expand on, is being closed off section by section and SQLedger went nowhere.

When it came time to look at this again Tiny ERP, now Odoo, stood out as a great option feature and customisation wise and it was now web-based but its business model left no doubt that the community edition is on notice. While looking we came across Flectra, a fork of Odoo, and it at least seemed to address some of the risks with Odoo. There was a lot of bad stuff being said online but since we were new to the Odoo/Flectra world it was hard to tell who was "in the right". Suffice it to say no one looked clean but I could understand the fork given Odoo's moves to reduce functionality. We wouldn't want to sign up for a project that looked hostile to its community user base.

The lack of a vibrant community around Flectra and that it was a new, unproven project was a big risk for us but we were under pressure and accepting the risk implemented Flectra.

Shortly after implementing Flectra 1.6 and then upgrading to 1.7 Flectra 2 was announced. This was not great news for us major version upgrade option from Flectra and Odoot is something we knew was going to happen sooner or later. Unlike Odoo where it is relatively easy to find assistance in the community with Flectra, this is less so. Flectra's main communication channel is Telegram which is a walled garden that cannot be queried nor searched with a browser. Questions need to be answered time and again as there is no history for users to refer to. It's hard to build up a knowledge repository this was which is supposed to be one of the benefits of open-source for adopters.

This journal is our contribution to the Flectra community and to FlectraHQ to see if we can help build up the expertise and knowledge that a vibrant open-source community can provide and to thank the Flectra developers for their efforts. I am not sure where this journey will end but I am hopeful that with the community expertise, we can assist each other and Flectra with this important project.to get all things sorted. To make sure its clear we have not yet completed the upgrade process but want to document what we have tried so far in the hope it helps others and we can work out the steps required for a successful upgrade as a community.

Base Server Setup

We set up a virtual machine to test the upgrade process. This would allow us to snapshot and roll back as needed as we experimented with the upgrade. We are using a Ubuntu 20.04 virtual machine as its version of Python and Postgres have fewer issues with Odoo 12,13 and 14 than Ubuntu 22.04. to get all things sorted.to get all things sorted.to get all things sorted.to get all things sorted.to get all things sorted. Our production server is using Ubuntu 18.04 just in case this becomes relevant later.

The plan is, based on fragmented information online, to upgrade from Flectra 1.7 to Odoo 12, then 13 and 14 before installing Felctra 2. The major version upgrades will be done with the Odoo community tool OpenUpgrade. There was a change in the way OpenUpgrade works from 13 to 14 which we will document, but the high-level steps for each version upgrade are as follows:

Install Odoo.x from deb packages
Install OpenUpgrade migration scripts for the relevant version (e.g branch 12.0 for Flectra 1.7 to Odoo 12)
Copy over and install the relevant data, and files to be upgraded
Update our custom modules
Install our standard modules for the Odoo.x version
Run the OpenUpgrade process
Run the Odoo.x update
Start the Odoo.x process and see if it works
Repeat till we at the right version for flectra2
Update to Flectra2
Profit!!!

Sounds like a walk-in-the-park right? Sadly far things don't look good. Here is what we have achieved so far.

Getting ready to go

Postgresql Setup

Install Postgres
- sudo apt install postgresql
Downloaded the database backup file and restore to the VM. In Postgres CLI:
- "create role flectra with login encrypted password 'flectra';"
- "create database finance with owner flectra;"
  - make sure to name the database the same as in production as it is used in file store paths.
- "pg_restore -h localhost -U flectra -W -d finance ~/Downloads/finance.dump"

Filestore Setup

Download the filestore under "/var/lib/flrectra/.local/share/Flectra/filestore". Probably best to create a tar file and copy that over.
Copy the filestore to /var/lib/odoo/.local/share/Odoo/filestore
- "sudo cp -a ./var/lib/flectra/.local/share/Flectra/filestore /var/lib/odoo/.local/share/Odoo/"
Change ownership to Odoo:Odoo
- "sudo chown -R odoo:odoo /var/lib/odoo/.local/share/Odoo/filestore/"

Note: The /var/lib/odoo/.local/share/Odoo/filestore directory will not exist until you run Odoo so you may need to create it or run odoo to create it then stop odoo and copy over the files.

Update and Install Custom modules

We don't know a whole lot about Odoo and Flectra internals and development but have written some basic modules. This processes is teaching us a whole lot about Flectra's internals we wish we didn't need to know but it all helps in the long-run.

Copy over your custom modules. We grouped them in an addons-upgrade directory.
- "scp -r user@hosting.co.za:/home/user/addons-upgrade ./"
Update references to Flectra to Odoo
- "grep -ilR flectra ./ | egrep -v 'git' | xargs -n 1 sed -i 's/flectra/odoo/g'"
- "grep -ilR 'fpl-1' ./ | grep -v git | xargs -n 1 sed -i 's/FPL-1/OPL-1/';"
  - this is to stop complaints about licenses during the upgrade
Finally, copy over the custom modules to the addons directory
- sudo cp -a ~/addons-upgrade/* /usr/lib/python3/dist-packages/odoo/addons/

Note: The addons directory will not exist until you install Odoo.

We had to edit "views.xml" , "templates.xml" and models to accommodate changes between versions of Odoo. This is something that will be different for your custom modules. Our modules are simple so it was mainly related to xpath queries to change the layout templates. We did find that, on occasion, we needed to delete the entries from ir_ui_view in order for changes to templates.xml to be loaded despite running an update.

"delete from ir_ui_view where id in (1554,2126);"

Install the version of Odoo you are upgrading to

Download Odoo
- wget https://nightly.odoo.com/12.0/nightly/deb/odoo_12.0.latest_all.deb
Install the python dependencies (you can probably skip this and fix it with the last 2 steps)
- "sudo apt install python3-docutils python3-feedparser python3-gevent python3-html2text python3-mock python3-ofxparse python3-passlib python3-psutil python3-pydot python3-pypdf2 python3-serial python3-suds python3-usb python3-werkzeug python3-xlsxwriter node-less python3-vatnumber python3-num2words python3-babel python3-decorator python3-jinja2 python3-mako python3-psycopg2 postgresql-client python3-ldap python3-pip pyldap qrcode xlwt vobject -y"
Install Odoo
- "sudo dpkg -i odoo_12.0.latest_all.deb"
- "sudo apt install --fix-broken"
  - this will resolve the broken install you will get from the step above
Edit /etc/odoo/odoo.conf as appropriate
- [options]
  ; This is the password that allows database operations:
  ; admin_passwd = admin
  db_host = localhost
  db_port = 5432
  db_user = flectra
  db_password = flectra
  ;addons_path = /usr/lib/python3/dist-packages/odoo/addons
Install these python modules that may be needed later
- "sudo pip install pyldap qrcode xl

Install OpenUpgrade

Next, we need to install OpenUpgrade. The OpenUpgrade library is required by the OpenUpgrade scripts. As far as we can tell this needs to be installed separately from the GitHub repository. Don't use the library that comes with Ubuntu 20.04 instead install the latest version with:

sudo -H pip3 install --ignore-installed git+https://github.com/OCA/openupgradelib.git@master

Next, we need to clone the github repo and switch to the right branch for the upgrade. In this case (Flectra 1.7 (Odoo 11) to Odoo 12)

git clone https://github.com/OCA/OpenUpgrade.git
git switch -c 12.0 origin/12.0

Install The Odoo Versions of Any 3rd Party Modules

We use the account financial report module amongst others. We didn't find versions of all the 3rd party modules that we used in Flectra. This is probably the cause of some of the issues we have experienced.

sudo cp account_financial_report-12.0.1.5.2.zip /var/lib/odoo/.local/share/Odoo/addons/12.0/
sudo unzip /var/lib/odoo/.local/share/Odoo/addons/12.0/account_financial_report-12.0.1.5.2.zip
sudo chown odoo:odoo

Ready, Set, Go

We will assume the steps to upgrade complete for now. As you may suspect, running the upgrades steps is where most of the work lies and we will document the challenges we came across in the following sections.

Stop the odoo service.
- "sudo systemctl stop odoo"
Run the OpenUpgrade script from within the OpenUpgrade directory.
- "sudo .Openupgrade/odoo-bin -c /etc/odoo/odoo.conf --data-dir /var/lib/odoo/.local/share/Odoo --database finance --update=all --stop-after-init"
Next run the Odoo Upgrade (Note we are running odoo11 not OpenUpgrade binary)
- "sudo odoo -c /etc/odoo/odoo.conf --data-dir /var/lib/odoo/.local/share/Odoo --database finance --update=all --stop-after-init"
Start Odoo
- "sudo systemctl start odoo"

Profit?

Not quite. As you can guess the upgrade steps don't go according to plan..

Issues Before the Upgrade

Before we began the upgrade to Odoo 12. We ran an update on Flectra 1.7 to make sure the database was up-to-date.

"sudo odoo -c /etc/odoo/odoo.conf --data-dir /var/lib/odoo/.local/share/Odoo --database finance --update=all --stop-after-init"

Felctra 1.7 Theme Update Errors

During this process, we got errors updating the themes. Theme Art, Hermit, Techperspective etc were noted as not upgraded due to being incompatible. We got the same error attempting to update them via the web interface. We tried uninstalling them and then reinstalling them. We managed to get Theme Art to install, at least according to the web interface. None of the other themes would install though. Strangely it didn't seem to bother the e-commerce that their themes were missing. We have multiple e-commerce sites and we cannot use the same theme on different sites and so had to install a different theme for each site. So before the upgrade only Theme Art was installed even though other sites where using "Theme Hermit" etc. So there is some issue here that may be causing issues with our upgrade. Anyone else had these problems with theme and Flectra 1.7?

Issues Upgrading from Flectra 1.7 to Odoo 12

We got several errors during the OpenUpgrade process.

The process halted with an error about an insert into the name column on the product_template table being null. The insert appears to be part of account_invoice migration. There is a line from a purchase invoice that the process wants to create a product template for. Its for a tip on a lunch bill. I have no idea why. There are two rows with this problem. We got around this by adding a default value to the column with:
- "alter table product_template alter column name set default 'change me'";
We got errors when the upgrade script tries to delete some lines from ir_ui_menu. The delete fails due to a foreign key on the menu_bookmark table. We delete the entries from menu_bookmark. It might be a good idea to just unbookmark everything in production before backing up the database.
- "delete from menu_bookmark where menu_id=365;" -> Your menu_id will be different.
We had an error when the process was trying to delete an line from the res_country table for Romania as it was referenced from res_partners. I found the account and updated its address to point to another country and will come back and switch it back once done.
- "update res_partner set country_id=27, state_id =339 where id = 2879;"
It complained about duplicates in the product_wishlist so I found and deleted those.
- "delete from product_wishlist where id in (620,572,334);"
For one of our modules we needed to update references to the class WebsiteSalesOption with WebsiteSale as this had been merged in 12.0

After this the OpenUpgrade process ran to the end but the following modules were not upgraded.

module base_branch_company
module account_bank_statement_import_ofx
module account_discount
module theme_art
module currency_rate_update
module account_asset_management
module sales_discount
module account_cash_flow
module payment_payweb_3

We will need to check the ir_module_module table to see if we can work out the modules we need here, mapping from Flectra to Odoo.

Running Odoo 12

I edited my client's host file to point to the upgrade server to ensure that any url would be mapped correctly and then started the Odoo service and access the website with http://url:8069

The first page looked promising and had only minor style issues but upon logging in numerous errors were presented. We suspect most of these relate to the missing modules.

When browsing to /shop we get 500 errors. The following is in odoo-server.log
- KeyError: ('ir.qweb', <function IrQWeb.compile at 0x7f96da9f59d0>, 'website_sale.product_view_switcher', ('en_US', None, None, None, False, 2))
- KeyError: ('ir.ui.view', <function View.get_view_id at 0x7f96d3866dc0>, 4, 'website_sale.product_view_switcher', (2,))
- ValueError: View 'website_sale.product_view_switcher' in website 2 not found
When browsing to /web we get a pop up with:
"Could not get content for /account_asset/static/src/less/account_asset.less defined in bundle 'web.assets_backend'."
Closing the above pop-up we get other errors depending on the menu item selected. Here are some of them.
- Inventory -> Master Data
  - ValueError: Field `asset_category_id` does not exist

We will be updating this post before documenting our process from 12 to 13. We are unsure if we need to resolve all these problems in 12 before upgrading to 13 and then 14. Can we just ignore the missing modules until we get to Flectra 2 where they will be available again?

We continued to upgrade from Odoo 12 to Odoo 13 and overcome some obstacles there but we still got a way to go to get to Flectra2 and see if it all works. We will document these in a later post too. Please share your stories and experiences.

Advancing Your Cybersecurity Career

2024-02-22T08:47:12Z

In my previous blog post, I set out our recommendations for starting out on your cybersecurity journey dealing with the fundamental skills required to become a cybersecurity professional. In this post, we start to look at specifically cybersecurity-focused certifications.

In the world of cybersecurity, obtaining certifications is one way to demonstrate your knowledge and expertise in the field. There are many different types of certifications available, but they can generally be divided into two streams: technical certifications and management and governance certifications. In this post, we will focus on management and governance certifications that are specifically geared toward cybersecurity professionals.

Although management and governance certifications require some technical knowledge, they tend to focus more on the business side of cybersecurity. Professionals who hold these certifications often work in management positions and are responsible for overseeing cybersecurity functions within an organization. These professionals tend to be paid more than technical cybersecurity professionals, as their roles require them to interact with business units to align cybersecurity with the organization's goals. This means these certifications are more widely recognized by non-cybersecurity personnel and the value the holders of these certifications bring is easier for them to identify and this, fair or not, translates into higher pay.

Here are some of the top cybersecurity management and governance certifications recommended by Jumping Bean:

Certified Information System Security Professional (CISSP): Offered by the International Information System Security Certification Consortium (ISC)², the CISSP certification is globally recognized and covers eight domains related to cybersecurity management. These domains include security and risk management, asset security, communication and network security, identity, and access management, security assessment and testing, security operations, software development security, and security architecture and engineering.
Certified Information System Auditor (CISA): This certification, offered by ISACA, is designed for professionals interested in auditing, control, and security of information systems. It covers topics such as information system auditing, governance, risk management, and information security management.
Certified Information System Manager (CISM): Also offered by ISACA, the CISM certification is designed for professionals interested in cybersecurity management roles. It covers topics such as information security governance, risk management, incident management, and program development and management.
Certified in the Governance of Enterprise IT (CGEIT): This certification, also offered by ISACA, is designed for professionals who are responsible for managing and governing enterprise IT. It covers topics such as IT governance frameworks, strategic alignment, and value delivery.
PECB 27001 - ISMS: This certification, offered by the Professional Evaluation and Certification Board (PECB), focuses on the implementation and management of an Information Security Management System (ISMS) based on the ISO/IEC 27001 standard.
PECB 27002 - Controls: This certification, also offered by PECB, focuses on implementing and managing information security controls based on the ISO/IEC 27002 standard.

It's important to note that these certifications have prerequisites and require ongoing education and recertification to maintain their validity. As with any certification, it's important to consider your career goals and choose a certification that aligns with those goals. In our next post, we will cover recommended technical certifications for those who prefer technical work.

Want a career in Cybersecurity?

2023-04-02T07:19:49Z

As artificial intelligence begins to automate many of the entry-level jobs in IT, the cycle of automation started by the first industrial revolution is disrupting a sector previously thought a safe haven. It has become more important than ever to find opportunities for knowledge workers where demand for skills outstrips supply and where machines cannot yet replace humans. One such sector is cyber security.

As a subject matter expert at Jumping Bean, I am often asked how one gets started in cyber security. What is the suggested learning path to acquire the skills necessary to be a cybersecurity professional? What cyber security courses should one attend?

To start a career in cyber security one needs a basic understanding of computing concepts, administering operating systems and an understanding of the fundamental network protocols. If you are completely new to the field I usually suggest doing a self-paced CompTIA A+, followed by a self-paced N+ course from CompTIA. CompTIA's A+ covers basic computing concepts such as hardware, operating systems and security. CompTIA N+ covers networking concepts in more detail and is essential for cybersecurity specialists. To get the knowledge to administer operating systems we suggest Linux Essentials from the Linux Professional Institute or Linux+ from CompTIA for self paced study .

It is not necessary to get the certifications but one should master the objectives of these certifications to make sure one has the right foundation for tackling cybersecurity. If you already have these skills, either from your previous job experience or tinkering with computers you may already have the foundation to start the process of becoming a cybersecurity specialist.

Foundational Skills & Knowledge for cyber security

The first cyber security-specific training course we suggest is CompTIA's Security+. This course is a basic cyber security course but will make sure your foundations are solid for undertaking more in-depth cyber security courses. Once this has been accomplished it's time to decide if one wants a more technical or managerial career path in cyber security. By this I mean does one want to manage the IT information security function, and ensure cyber security risk is addressed in a compliant way with appropriate organisation governance structures and procedures or does one want to be more technical and be a Red/Blue team member that undertakes penetration tests, test control compliance and configure secure infrastructure?

Tune in for more

Tune in to the second part of this series where we cover the managerial certification path and in part 3 we look at the technical certifications. We will update this page with links once they are published.

Quiz Time - 5 Jan 2023

2023-01-06T09:46:58Z

The possibly unexpected answer to the above is C. This is one of those trick questions they like to ask in certification exams or interviews. To understand why the answer is C and not A one needs to appreciate what the operators do. This type of "trivia" is often glossed over when one is learning Java but having a solid understanding of what is going on turns one from a mediocre programmer to a great programmer. Our Java Essentials course makes sure we cover all these bases.

Post Increment/Decrement Operators Sequence

The post increment operator (x++) will return the value of the operand then increment it while the pre increment (++x) will increase the value then return it.

So by the time we get to "return x++", the original value of x = 10 would have been reduced to 9 by the "x--" statement. This statement would have reduced x to 9 and then returned 9 but it wasn't assigned to anything but x was updated. So when we get to "return x++" it first supplies the value of x which is 9 to the return statement and then increments x. Incrementing X no effect as function has returned and the x parameter is now out-of-scope.

Incomplete Explanation - Operator Precedence

Often people refer to operator precedence when dealing with this question but the post-increment/decrement and pre-increment/pre-decrement operators have a very high precedence and therefore this doesn't explain the observed behaviour.

Java Operator Precedence Table

Operators	Precedence
postfix	`expr++ expr--`
unary	`++expr --expr +expr -expr ~ !`
multiplicative	`* / %`
additive	`+ -`
shift	`<< >> >>>`
relational	`< > <= >= instanceof`
equality	`== !=`
bitwise AND	`&`
bitwise exclusive OR	`^`
bitwise inclusive OR	`\|`
logical AND	`&&`
logical OR	`\|\|`
ternary	`? :`
assignment	`= += -= *= /= %= &= ^= \|= <<= >>= >>>=`

Pacman Proximity Alarm

2023-01-06T09:40:17Z

Ever since Arduino first captured the imagination of the world I have harboured the desire to learn and master the art of hardware hacking. Actually, even before the Arduino, I wished to know more about the devices that I programmed, almost exclusively microprocessors, and to be able to create new solutions using hardware and not just software. My thought, at that stage, was that I would first need to learn electronics and electrical theory before being able to start building circuit boards.

Arduino's Abstract Away the Engineering Behind Circuit Boards

This frame of mind made it a bit difficult to understand the Arduino when I first approached it since it requires only a basic understanding of electronics to start building new and exciting hardware solutions with it. Now I appreciate, as probably most others did right away, that the Arduino abstracts away the complexity of printed circuit boards and provides one with a Lego-like array of sensors and actuators to design and build your own hardware solutions.

The key to conceptualizing and implementing your own creation is to know what type of sensors exists, what they can do and how they work, and then imagine new ways to combine them to create something that solves a problem for you.

Perhaps someday I will still get down to designing my own printed circuit board but the Arduino is a great halfway point on that journey.

The Vision

For a while, we have had a Pac-man Ghost lamp at the office that was relocated from its position at reception when we set up our POS terminal and has since been lying around idle. It is powered by a USB connector and the thought crossed my mind it would be great to wire it up to an Arduino, throw in a distance sensor and have it light up when it detected someone moving around the reception area. Besides being cool, at least for me, it would also serve as a notification to everyone that there was some activity at reception.

To fully appreciate the vision you need to know that we have a Pac-man-themed floor at the office.

Components Diagramme & Sketch Code

Below is a schematic diagram of the final design. It's my first time using the fritzing layout software so please forgive the skew lines. I couldn't work out how to make the connectors align correctly.

Below is the list of components used. These were selected because that's what I had on hand.

Arduino UNO
Ultra-Sonic SR04 Sensor
Pezeiro buzzer
Pac-man Ghost lamp
Old Micro USB cable

Below is the sketch code for the Ghost Lamp Proximity Alarm

#define trigPin 2 #define echoPin 3 #define lightPin 11 #define buzzer 4

int lastDistance=0;

void setup() { pinMode(lightPin, OUTPUT); pinMode(trigPin,OUTPUT); pinMode(buzzer,OUTPUT); pinMode(echoPin,INPUT); lastDistance=getDistance(); delay(50); }

void loop() { int currentDistance = getDistance(); //want to try minimize false alarms so only sound alarm if distance changes by 2cm // and only sound alarm if object is approaching not redecing if (lastDistance-currentDistance>2){ soundAlarm(); }else { shutoffAlarm(); } lastDistance = currentDistance; delay(50); }

int getDistance(){ digitalWrite(trigPin, LOW); delayMicroseconds(5); digitalWrite(trigPin, HIGH); delayMicroseconds(10); digitalWrite(trigPin, LOW); int duration = pulseIn(echoPin, HIGH); return duration * 0.034 / 2; }

void soundAlarm(){ tone(buzzer, 1000, 500); digitalWrite(lightPin, HIGH); }

void shutoffAlarm(){ digitalWrite(lightPin, LOW); }

Step-by-Step Guide

I approached this project just like one approaches a coding task. Identify the sub-components, get them working, then integrate them and test them along the way.

The three main tasks consisted of:

the Ghost Lamp - providing it with power and getting its LED to fire with code
the Proximity sensor - wiring it up, getting the distance calculations correct and integrating it with the Ghost Lamp firing code
the buzzer

The Ghost Lamp - Wiring up the USB to the Ghost Lamp

I started with snipping off the USB A connector from the USB cable. This caused me some distress as I hate destroying something that is working fine and has a useful life ahead of it.

I would have preferred to use one of those faulty USB cables that I have lying around somewhere but as Murphy knows, it's only locatable when you need a working one.

A USB cable is simple, 4 wires, 2 for sending and receiving data and 2 for ground and power. I deduced that the Ghost was not using the data transmission wires. Once stripped the 4 wires were easy to identify by their colours. Red for the 5v wire and black for the ground. The hardest part was stripping the thin wires to expose the core. Took me a few goes but I eventually got it right.

To connect the exposed wires to the Arduino UNO I soldered some male-to-male connectors I had lying about to the wires. I attached the connectors to the UNO's ground and 5V pin for a quick test. Success! The Ghost lamp lit up!

Next was to write up a quick sketch to pulse the power on one of the digital pins, rewire the live wire to the selected pin, compile and test. Again success! This is essentially the same sketch one uses in the Hello World of Arduino programming but instead of a blinking LED, I had a blinking Ghost light.

The Ultrasonic SR04 Sensor - basic distance sensing algorithm

Next was the Ultrasonic SR04 sensor. There are several types of distance sensors. The SR04 uses ultra-sonic sound waves bouncing off objects to detect their distance whilst other sensors use infrared like the infrared obstacle avoidance sensor. The IR distance sensors work with infrared and reflected light instead of sound waves.

Wiring up the Ultrasonic SR04 sensor wasn't too difficult. Like most components, it has a VCC pin and ground pin. The magic happens with the trigger and echo pins. The trigger pin initiates an ultra-sonic burst and the echo sensors detect when the signal bounces back. Using the time elapsed between triggering the ultra-sonic burst and receiving its echo and the speed of sound one can calculate an approximate distance to the nearest object. For the technical details on the SR04 Ultrasonic sensor and the timings used in the code check out the SR04 datasheet.

In a complex set-up, like my workspace, this was problematic as the sound waves returning to the sensor may be bouncing off some static foreground object instead of a more distant, larger moving object I am interested in.

Like if you are trying to detect someone moving closer to the sensor in a room, and its distance from the sensor, rather than just what is the nearest object in the room. At my workspace, the sensor would register a change in the distance occasionally even if no object was moving. This could be due to sound waves bouncing off objects and causing interference and arriving at odd times etc. But I am just speculating here as I am not a sound wave expert. Maybe it was detecting those evil IT gremlins that cause so many unexplained technical issues? I will test in a less complex space over the next couple of days and let you know.

The Pezeiro Buzzer

Finally, I added a Pezeiro buzzer. I hoped to be able to implement the waka-waka Pac-man sound effect but sadly my Google foo failed to find any existing code for this nor could I find any musical note notation for it. I could find the notes for the Pac-man theme tune but that is way too much typing to implement. I settled for simply sounding the buzzer when movements were detected for now. This was easy enough to integrate into the code.

Deployment

I deployed the set-up at the office. It works better than at my crowded workspace but still gives off some false alarms. After a while, the buzzer was disconnected for obvious reasons.