Arch Build System

Arch Build System (ABS) is Arch’s way of streamlining your workflow when you need to build a custom open source package.  It is well-documented on Arch Wiki here – https://wiki.archlinux.org/index.php/Arch_Build_System.

Here’s a tl;dr example-oriented summary to help new Arch Users get right down to it.  This example demonstrates how to build your vim editor with python-support enabled.

# Install and run abs (sync)
sudo pacman -S abs
sudo abs
# Prepare a build area
mkdir ~/abs
cd ~/abs
# Make a local copy
cp -r /var/abs/extra/vim .
cd vim
vim PKBGUID
# Change the "--disable-python" options to "--enable-python"
# Build the package
makepkg
# Wait for a while
# Install your new vim and runtime (these are created via the makepkg command above)
sudo pacman -U vim-runtime-7.4.335-4-x86_64.pkg.tar.xz
sudo pacman -U vim-7.4.335-4-x86_64.pkg.tar.xz
# Check it for "+python"
vim --version

Finally, if we are going to make sure that future updates to the vim package with pacman does not override our custom vim.  To do so, simply declare

# Pacman won't upgrade packages listed in IgnorePkg and members of IgnoreGroup
IgnorePkg   = vim
#IgnoreGroup =

And that’s pretty much it.

The important thing to remember is that your custom build depends on the PKGBUILD file.  The `abs` command rsyncs the PKBUILD file from arch repositories to your server whenever you need an updated PKGBUILD and this default PKGBUILD can then be copied over to your build directory and modified so you can decide which features you want to include in your customized package.

There are many community contributed custom packages in AUR (https://aur.archlinux.org/) or on github.com so we can use those too, if a PKGBUILD that enables python support is already maintained by another contributor.

Bottomline?  Everything that controls how a package is built is declared in PKGBUILD so getting familiar with PKGBUILD will let you manage any custom package build you want.

Why scrum? Why agile development?

Scrum and Agile development are simple methodologies intended to solve the problem of

  • long product development cycles; and
  • a mismatch between a product’s business requirement and the actual resulting implementation

This illustration explains such mismatched expectations, and the consequence of long development cycles.  For anyone who has been working in the software development industry for a while, you would probably have gone through the hell of the typical waterfall development approach.  With an outcome where everyone in the team is demoralized and where you end up with a bunch less-than-happy customers – an unfortunate, dysfunctional outcome.

mismatched_expectations

Agile product development has its roots from lean manufacturing (not to be confused with lean start-up) from Toyota; and is a general framework to guide small, empowered teams to execute on their product development work in an iterative fashion.  The Scrum Master course I attended in Feb talks about a specific practice in agile product development known as “Scrum“.

Shorter development cycles (called “sprints”) of 2 to 4 weeks and daily team meetings known as “scrums” allow the product team, the product owner and the customers/stakeholders to regularly review the incremental improvements made by the team in each sprint.  This narrows the expectations gap significantly and the increased communication level between the product team and the stakeholders reinforces a shared vision for each product increment made.

The ideal size of a product scrum team ranges from 3 to 9 (but 9 is already too big in my my opinion), comprising sales/marketing and design/product specialists in each team, with a typical ratio of 1-2 designers/marketers to 5 developers in each team.

The 2-4 weeks sprint cycle can be visualized with this diagram.

scrum-overview-mark-hoogveld

Some definitions:

Product Owner

In each team, there will be a “product owner”, also sometimes referred to as the “product manager”.  This is the person responsible for prioritizing product development features in each sprint and over the course of the entire product development (a product development process comprises of many many sprints, each sprint being 2 weeks to *maximum* 4 weeks).

The Product Owner is almost always the business stakeholder and is directly responsible for the financial well-being of the project, including profits/revenues when the product is launched to customers.

Scrum Master

For a team of 5-7, it is also the norm to have a Scrum Master.  Note that the roles of the Scrum Master and the Product Owner are completely different things.  The Scrum Master is a guide and a process-people-skills-resource facilitator.  The Scrum Master, however, *does not* set the product direction and is not the person to decide on product feature priorities.  It is the job of the Product Owner/Manager to decide on product feature priorities.

It is important for the roles of Scrum Master and Product Owner to be cleanly decoupled – two separate individuals.

Product Backlog

The team and the product owner comes together on the first day (at the start of a sprint cycle) for a 2-4 hours meeting to define the product vision and to list out all the product features that needs to be developed based on the product vision (also known as the product “hypothesis” per lean-start-up concepts).  The outcome of this 2-4 hours meeting is a long long laundry list of features that should be build that are hypothesized to serve the product vision/goal.  This laundry list is known as the “Product Backlog”.

Sprint Backlog

After the Product Backlog has been established, the next 2-4 hours is spent on planning for the *current* sprint (I will simplify things by saying that each sprint is 2 weeks, not 2-4 weeks).   Many of the Product Backlog items will be kicked out and only items that *The Team* can comfortably complete in the current sprint (2 weeks) will be placed into the “Sprint Backlog”.

Once the Sprint Backlog has been decided, the sprint commences on Day 2.  Notice that Day 1 is almost always spent on building team clarity and deciding on feature prioritization with *everyone* present and involved.

In an ideal scenario, it also makes sense to invite “Customers” and “Users” of the product that is being developed to participate in a limited way during this first day of planning.  This is where “Lean Start-up” practices meet “Scrum/Agile Product Development”.

Daily Scrum

Once the sprint begins proper, a 15 minutes session is allocated at a fixed time *EVERYDAY* for the team members, scrum master and product owner to do a stand-up meeting.  4 key points are covered by the product owner and the team in this 15 minutes.  Each member will summarize and share:

(1) Done: What has been done by him/her

(2) To Do: What needs to be done next by him/her

(3) Blockers: any issue (whether Technical or Administrative or Human-related) that is preventing him/her from achieving his/her personal tasks

Finally,

(4) Updates & Re-prioritization, if any:  The Product Owner can provide market updates/discussions with meetings with clients on the previous day etc to the team.  The Product Owner can also decide to *CHANGE PRIORITIES* of items that needs to be done.  When certain development item’s priorities get changed, it will – in all likelihood – result in existing features being de-prioritized.  The sprint timebox is strictly 2 weeks (for newer teams, it is preferable to work in 4-weeks time frame to factor in time for picking up new knowledge and getting to know how each other operates).  Which means like the “law of conservation of energy”, what gets added in implies something else gets kicked out (and thus, gets removed from the Sprint Backlog and placed back into the Product Backlog).

Benefits

There are a few advantages for having a daily scrum.  For example, good development work requires engineers to be in the zone when they are working.  Instead of “interrupting” a developer who is deep in his/her code and working through his/her current task, by having a fixed timing for the 15 minutes daily team update,  Product Owner and Scrum Master do not “break the developers’ zone”.  Breaking the zone is a huge source of lowered productivity and causes increased mistakes in software development.

On the other hand, *not* having regular communications between the product owner and the team will result in mismatched expectations and poor implementation prioritization.  So having this regular 15-minutes communication on a daily basis is the basis for team to collaborate optimally.  Market/customer updates brought in by the marketing specialist or Product Owner during this 15-minutes scrum meeting is critical for ensuring that everyone is “in-sync” with and “re-synced” to what needs to be delivered.

Scrum” is simply a formalized term to describe a set of practices for a professional product development team to work well together.  The focus is on keeping collaboration processes simple and organized. It is not a “magical” cure-all solution.  It still requires that the company hires the right people with the right aptitude and attitude for the team, and that the team uses common sense to work with each other, and more importantly, that the product we are developing finds the right product-market fit (lean start-up).  A product team can be top notch, but if the product fails to gain market acceptance, we are still doomed.

For a start-up product (or more generally, software project) to be successful, it needs to have BOTH market traction and excellent product execution, product execution using “Scrum” and the agile development methodology as explained in this post.  And market traction, as a completely separate topic and as a story for another time, depends on the ability of the product owner and founders of the start-up to determine market size and the market need for their start-up product.

Interactive web clients: frontend routing or backend routing?

As frontend web frameworks like AngularJS, BackboneJS and EmberJS (and ReactJS, though ReactJS doesn’t give you MVC, only V) gain prominence, a question that’s often asked and explored is — where do we specify our routing?

Frontend MVC-style JS frameworks usually come with libraries that help you define url routes.  And frontend MVC-style JS frameworks are often coupled with NodeJS, where NodeJS can act as the backend server to provide you a way to specify your url routes like Django’s urls.py does or Rails’ routes.rb or martini-go’s m.[http method GET, PUT, POST, DELETE, PATCH] do.  Typically, if we implement backend routing in NodeJS, it will be represented as such:

// ... more code above, truncate code here

var app = connect()
     .use(express.static(__dirname+"/build"))
     .use(express.static(__dirname+"/bower_components"))
     .use( function(req, res) {
var pathname=url.parse(req.url).pathname;
var localpath = path.resolve(__dirname);
switch(pathname){
    case '/register':
        getfile(__dirname+"/html/register.html", res, "text/html");
        break;
    case '/homepage':
        getfile(__dirname+"/html/homepage.html", res, "text/html");
        break;
    case '/upload':
        getfile(__dirname+"/html/upload.html", res, "text/html");
        break;
    default:
        getfile(__dirname+"/html/index.html", res, "text/html");
        break;
}
});

http.createServer(app).listen(8000, serverUrl);

which as you can see, is fundamentally analogous to the framework interfaces provided by python/django or python/flask or ruby/rails or golang/martini-go.

In short, if we so choose to use nodejs for backend routing when implementing our web client, we are essentially asking our server backend for url route definitions.

This dilutes the benefits associated with implementing a frontend-only web client because we will in fact be polling our server for a URL whenever we render a new “page” (screen).  Put it another way – would you implement an iOS or Android app that needs to ask the server which screen to transit to whenever a user needs to load a new screen?

Viewed in this context, the answer is clear — we should be using frontend routing to leverage on the full benefits provided by feature-complete MVC JS framework like AngularJS or BackboneJS or EmberJS.  This means that the end user of our web app only loads our html/js/css files once.  Dynamic data on each “screen/page” is retrieved from a REST API and has nothing to do with the “screens/pages” already implemented in the frontend, which is why all routes are *already decided* by our frontend routing the first (and-only) time your user’s web browser loads up the site.  This is also why we often refer to these type of web apps as “single page app”.

Because of frontend routing via these JS frameworks, we will see a strange hashbang symbol in our url, like this http://ourdomain.com/#/login .  Such urls are not cool in my books and I want my pretty urls!  So sad… fortunately for us, HTML5’s pushstate feature is readily available and already implemented in all these modern MVC JS framework. With a little configuration, we can have our pretty urls without the hashbang-ed urls.

Like this:-

AngularJS – http://scotch.io/quick-tips/js/angular/pretty-urls-in-angularjs-removing-the-hashtag

BackboneJS – http://zachlendon.github.io/blog/2012/02/21/backbone-dot-js-from-hashbangs-to-pushstate/

What about my SEO?

The astute reader and seasoned web developer/owner will ask a question at this point — if it’s a “single page app” that is fundamentally rendered by hashbang urls, wouldn’t our SEO be screwed?

Great question, but developers of these frameworks are seasoned web veterans…

Handling SEO is nothing but providing a way for search engine crawlers (“bots”) to navigate your website and associate content/keywords on each “url” on your website so that search engine users can find your content. Here’s a great article detailing how to “serve” content to the bots while providing the seamless experience provided by a single-page app to the human user – http://backbonetutorials.com/seo-for-single-page-apps/

Long story short, we will *still* deploy a nodejs app via phantomjs with generated “backend routing” solely to serve our bots.  As far as our human friends are concerned, they will never need to access these content or poll the server to figure out url routes.  Best of both worlds.