Woohoo, haven't had time to post a contribution thread in ages...
I finally got around to fixing the upload scripts for my site and ended up completely rewriting the content management system. The old one was a Frankenstein's monster of bash, perl, pandoc, php and about 5 metric tons of sh..., um, kludges.
The new Xac is much simpler. It's all written in (relatively clean) Python 3. It takes files from different source directories, processes markdown through pandoc, interpolates some files if you want, and publishes everything to another directory that you can serve. The look and feel is managed through Pandoc output templates and can be XHTML, PHP, etc. It supports arbitrary plugins via simple Python functions (which can be provided by modules for more complex functionality).
The site itself is the example, and a working default is provided in the source files. If you want to check it out, start by downloading the source file and reading the README to get the server up (it basically just tells you to run serve.sh). You can view the default server's index page online too. The default server is admittedly a bit complex, but that's because it is what I actually use for the site (although I have some custom functions for managing my repos and other things). You can layer any level of complexity on top of the base functionality because it's all in Python (i.e. not an arbitrary configuration language), but you can keep it dead-simple too. The source archive includes a minimal example too.
I'll continue to improve code and documentation as time goes by. If you have any questions, just ask.
Before anyone asks why they should use Xac, read this.
Incidentally, the site feels much snappier now with all that nasty PHP gone.
Hi, I'm toying with the idea of substituting my WP blog/site with xac. I was wondering if you have an example of a minimal blog setup. Basically, posts with the possibility of comments. If you don't have a worked out example, but can provide pointers that might help me develop it, please let me know too.
The default server in the source archive includes a simple news feed example. I use the same functionality for blog posts on my site. It may be a bit too simple for what you want, but if you know Python then you can use the news feed plugin module to create your own fully functional blog module. If you have specific questions I will try to answer them.
Btw, the current news/blog plugin auto-archives articles after a given interval (you can see it on my site). I intend to update it with an option to always keep at least x items in the main directory.
Comments could be implemented by using a custom template. For example, you could create a Pandoc template named "blog.php" (in "files/templates/") with support for a database-backed PHP comments section. Check the Pandoc documentation for information about Pandoc templates (example: files/templates/main.html). You can control which template file is used for a given path with the "get_pandoc_template" parameter (example: publish_simple.py).
You will probably need to insert a database ID into each template for the code to work. You can do that with Pandoc variables and "get_pandoc_args". The basic idea is simple. You can pass in additional variables with "-V foo=bar" to have "$foo$" replaced with "bar" in the template file during publishing. The simplest example is in "publish_simple.py", which returns no additional arguments and no input. A somewhat complicated example can be found in "Xac/defaults.py", which supports changing the template banner, the upper and lower navigation links, the footer, etc.
You could also embed an object to manage comments in an HTML template, instead of using a full PHP template. I don't know what comment systems are available though, but I expect that there is something generic. I've even considered hosting a FluxBB forum on the site and embedding threads on pages, but that might feel a bit too kludgy.
Last edited by Xyne (2012-07-16 23:39:41)
Thanks a lot, that was useful. I think I'll just start trying to migrate and bug you here if I have questions. Let's see what happens!
Barrucadu, thanks for the suggestion. I've been looking into it for a while and, indeed, implementing a commenting system is far from trivial.
The thing with Disqus is that precisely the whole idea of moving away from WP is to avoid bloat, and to avoid providing third parties with information about my behaviour or the behaviour of visitors to my site (which, otoh, is a very humble blog; hardly slashdot.org).
implementing a commenting system is far from trivial
Indeed. That's why I still haven't done it. Storing and retrieving comments is easy enough, but authentication, email verification, spam filtering, etc all require some thought.
I expect that there is some easy stand-alone comment system available somewhere than can be plugged in without much hassle, but I have never really looked for one.
I'm definitely interested in this and am looking forward to what you come up with. I may even end up trying some things myself.
Btw,in the latest version of Xac, after running publish.py the side bar appears at the bottom of the page. I'm trying to understand how it all works, and it'd help me if you could take a quick look and post where the bug is (if it's a bug)
When I run publish.py in the current release, I get this: screenshot
I re-released the source archive and double-checked it, so I am quite sure that nothing in Xac has changed.
The layout is controlled by the template and CSS files. Have you changed those? The default templates use a somewhat intricate hierarchy of nested div elements to achieve the layout, so even a very small change in the HTML or CSS can have a prominent affect.
Please post a screenshot if you can so I can see what you see. If you want some help editing the template files then pastebin what you have changed (or email it to me) and explain what you are trying to do.
If there is something else that is unclear about Xac, just ask. The questions will probably show me where the documentation needs to be improved.
Please find here three different screenshots showing how the test site looks like here after running publish.py twice -- with no changes in what ships in the tar file. The browser is dwb, but I've also tested it on firefox.
So far, I'm just trying to understand the test site. Once I have a clearer idea of what I'm doing, I'll start editing and, for sure, I'll take advantage of your generosity and bug you with my doubts
Even with dbw I can't reproduce it: dwb screenshot
I tested with several other webkit browsers on a different computer too.
How does the default site look before you run the publish script?
What about my site?
If those look fine then maybe you're missing one of the optional dependencies. The default server showcases several plugins. To use a plugin, you insert a CDATA-like block with named tags into the markdown. The plugin parses that block and replaces it with something else (e.g. a graph). If the plugin isn't supported then that data may get passed through as is. In some cases it might interfere with the HTML.
Can you pastebin or email me the generated pub/index.html page?
Btw, while I was debugging, I noticed that the instructions on the news page were outdated. I have updated it and removed unnecessary scripts from the archive. No extra scripts are necessary to manage the news section.
I have also added support for keeping a given number of entries in the main news directory.
You are right, I didn't have source-highlight installed. I had assumed, wrongly, that that could not be messing with the position of the side bar. Now it looks as it should.
Thanks for the support for keeping a number of entries in the news.
I had assumed, wrongly, that that could not be messing with the position of the side bar.
It was a reasonable assumption. I have updated the code to escape raw data blocks when a plugin fails. I actually had escaped the outer tags but not the inner data itself.
It's good to have someone else trying things with the code. I only use it in a very limited number of ways so I miss many cases. Thanks for the feedback.
New plugin: Aha
I've been looking into this again, and I have some questions. The first one: if I want to have a template specifically for news, I should change defaults.py to something like
def get_pandoc_template(xac, path): """Return path to template file for given path.""" if path == 'news': return os.path.join(TEMPLATES_DIR, 'news.html') else: if path: return os.path.join(TEMPLATES_DIR, 'main.html') else: return os.path.join(TEMPLATES_DIR, 'listing.html') return get_pandoc_template
Right? The code I have posted doesn't seem to work. Am I missing something?
Last edited by manolomartinez (2012-08-09 04:57:05)
Sorry for the late reply.
defaults.py should not be changed. It is part of the package and should be viewed as a static module. It is included both as an example of more complicated things that you can do with Xac, and as a convenience for me as it allows me to generate my site on different systems.
The path received by the function is the full server path, so "== 'news'" will never match. I would use the following function:
def get_pandoc_template(xac, path): """Return path to template file for given path.""" if path: if path.startswith('/news'): return os.path.join(TEMPLATES_DIR, 'news.html') else: return os.path.join(TEMPLATES_DIR, 'main.html') else: return os.path.join(TEMPLATES_DIR, 'listing.html')
Notice that I have also omitted "return get_pandoc_template" at the end. "get_default_get_pandoc_template" in defaults.py returns a function that returns a path. Here you are directly defining the function that returns the path, and you should define it in your own publish.py file (or in a separate file that it imports).
I'm looking for a simple CMS for my personal website and I really appreciate some of the features Xac offers, i.e. the directory listing.
Unfortunately running publish_simple.py returns me an error I don't understand:
[cem@amd64box xac-2012.12.26.1]$ ./publish_simple.py Traceback (most recent call last): File "./publish_simple.py", line 39, in <module> get_pandoc_args = get_pandoc_args, File "/home/cem/Setups_amd64box/xac-from-Xyne/xac-2012.12.26.1/Xac/publish.py", line 167, in __init__ self.watchlist_mtime = max(os.path.getmtime(x) for x in watchlist_files) File "/home/cem/Setups_amd64box/xac-from-Xyne/xac-2012.12.26.1/Xac/publish.py", line 167, in <genexpr> self.watchlist_mtime = max(os.path.getmtime(x) for x in watchlist_files) File "/usr/lib/python3.3/genericpath.py", line 54, in getmtime return os.stat(filename).st_mtime FileNotFoundError: [Errno 2] No such file or directory: '/home/cem/Setups_amd64box/xac-from-Xyne/xac-2012.12.26.1/<frozen>'
I don't think I'm missing any dependencies cause publish.py does work! (Exept for the tex2png plugin that returns me the same error mentioned in the AUR. I tried to rebuild my formats but couldn't get it to work till now. However that's not so important to me cause I don't need it at the moment, but it's also a nice feature.;)
I managed to publish a simple test page using the default publish.py. I'll try to continue with this one and see where I can get.
@Xyne: Thanks for sharing this anyways.
I was able to reproduce it but I'm still surprised. It seems that some Python module is reporting its path as '<frozen>'. I didn't bother investigating it further and simply added a try/except block to handle such errors. Please try xac-2013.1.
I have added a new plugin for inserting file content and a new function to the source highlight plugin to highlight external files. The plugins section of the default server index page has been updated with usage examples.
Looks great, I am finding more and more uses for Markdown by having Pandoc.
I've been using Hakyll, which seems to have similar functionality, to set up my blog. I would be interesting in any comparisons, beside the obvious of having to learn some Haskell (which has been rewiring my brain).
I haven't used Hakyll so I can't offer any comparison. I take it that you've already read the default server index page for an overview of usage and available plugins. If you download the source archive you can try the default server directly and take a look at the code.
I need to sit down and provide thorough documentation when I have some time. The publishing scripts for the default and the simple server are both commented and provide good starting points for customization. The core Xac module (Xac/publish.py) may be difficult to follow in some parts but normally you shouldn't need to touch that code (that's where the markdown conversion, file interpolation and file passthrough happens). The plugins are mostly self-documenting and some of them are very simple. The code for the default server is a bit hard to follow in some places due to lack of documentation as well. It nevertheless gives you some ideas of what you can do with Xac and reminds you that it's all just Python so there are no arbitrary limitations.
If you have any questions about the code or how to do something in particular, please ask. I will try to gradually add comments to the code in response to feedback as well.
As i was looking at the default index page I noticed that there were errors in the CDATA-like block escapes which likely caused some confusion. Sorry about that. The mistakes have been fixed in the last update.
The new file insertion plugins now support the path "%SELF%" for inserting a markdown file into itself.
Thank you for Xac, that's a very interesting project.
I wonder if it's possible to hide some folders in the folders list (eg. hide “etc/”).
Thanks for the feedback.
It's possible but a little difficult if you are using the default site configuration. The nav bar entries are determined by the get_pandoc_args function defined inside the get_default_get_pandoc_args function in Xac.defaults.py. The arguments to that function include the files and directories that will appear in the nav bar. If you define your own version then you can filter "etc/" there, but I admit the function is a bit chaotic and unpleasant to work with (the default site basically remains a hackjob). The process would be to copy that code into your own publish.py file and edit it as necessary, then pass your custom get_pandoc_args function to Xac.Xac.
If you are interested in using Xac then I recommend waiting a bit before you start hacking on it. I'm currently nearing the end of a complete rewrite that significantly improves speed, modularity and configurability*. I hope to release it soon (within the next 2 weeks) but as it progresses I keep getting ideas of how to improve it. I actually have the code open right next to the browser as I'm typing this.
* The rewrite also changes the implementation language (Python -> Haskell with Lua configuration files) as well as the plugin syntax, so migration will require intervention.
* The rewrite also changes the implementation language (Python -> Haskell with Lua configuration files) as well as the plugin syntax, so migration will require intervention.
Realizing I'll probaly be prosecuted for necro-bumping, and that Xac is generously shared with the community by Xyne, how do you handle that on a haskell virgin system:
Dependencies: haskell-pandoc (...)
... resolving dependencies... looking for conflicting packages... Package (87) New Version Net Change Download Size community/cmark 0.26.1-1 0.60 MiB 0.13 MiB community/ghc 8.0.1-1 1146.92 MiB 86.29 MiB community/haskell-aeson 0.11.2.0-10 17.18 MiB 1.29 MiB community/haskell-asn1-encoding 0.9.4-1 2.06 MiB 0.23 MiB community/haskell-asn1-parse 0.9.4-8 0.36 MiB 0.05 MiB community/haskell-asn1-types 0.3.2-5 1.96 MiB 0.19 MiB community/haskell-async 2.1.0-2 1.06 MiB 0.11 MiB community/haskell-attoparsec 0.13.0.2-5 13.90 MiB 1.01 MiB community/haskell-base64-bytestring 18.104.22.168-6 1.37 MiB 0.11 MiB community/haskell-blaze-builder 0.4.0.2-2 1.46 MiB 0.14 MiB community/haskell-blaze-html 0.8.1.2-1 31.39 MiB 1.23 MiB community/haskell-blaze-markup 0.7.1.0-1 2.79 MiB 0.21 MiB community/haskell-byteable 0.1.1-3 0.22 MiB 0.04 MiB community/haskell-case-insensitive 22.214.171.124-1 0.76 MiB 0.09 MiB community/haskell-cereal 0.5.3.0-1 5.08 MiB 0.40 MiB community/haskell-cmark 0.5.3.1-1 3.45 MiB 0.25 MiB community/haskell-conduit 1.2.7-1 5.43 MiB 0.43 MiB community/haskell-connection 0.2.6-1 1.03 MiB 0.12 MiB community/haskell-cookie 0.4.2.1-2 0.69 MiB 0.09 MiB community/haskell-cryptonite 0.19-1 24.29 MiB 2.05 MiB community/haskell-data-default 0.7.1.1-3 0.17 MiB 0.03 MiB community/haskell-data-default-class 0.1.2.0-1 0.44 MiB 0.05 MiB community/haskell-data-default-instances-containers 0.0.1-13 0.17 MiB 0.03 MiB community/haskell-data-default-instances-dlist 0.0.1-16 0.14 MiB 0.03 MiB community/haskell-data-default-instances-old-locale 0.0.1-13 0.14 MiB 0.03 MiB community/haskell-digest 0.0.1.2-3 0.28 MiB 0.04 MiB community/haskell-dlist 0.8.0.1-1 1.00 MiB 0.09 MiB community/haskell-enclosed-exceptions 1.0.2-2 0.28 MiB 0.04 MiB community/haskell-exceptions 0.8.3-1 1.89 MiB 0.15 MiB community/haskell-extensible-exceptions 0.1.1.4-14 0.17 MiB 0.03 MiB community/haskell-filemanip 0.3.6.3-6 1.48 MiB 0.16 MiB community/haskell-haddock-library 1.4.2-1 12.54 MiB 0.73 MiB community/haskell-hashable 126.96.36.199-5 1.07 MiB 0.10 MiB community/haskell-highlighting-kate 0.6.2.1-2 92.14 MiB 6.21 MiB community/haskell-hourglass 0.2.10-2 8.20 MiB 0.62 MiB community/haskell-hslua 0.4.1-3 3.54 MiB 0.32 MiB community/haskell-http 4000.3.3-6 8.32 MiB 0.78 MiB community/haskell-http-client 0.5.3.1-1 8.32 MiB 0.76 MiB community/haskell-http-client-tls 0.3.0-8 0.28 MiB 0.05 MiB community/haskell-http-types 0.9.1-4 3.17 MiB 0.27 MiB community/haskell-juicypixels 188.8.131.52-1 31.50 MiB 2.58 MiB community/haskell-lifted-base 0.2.3.8-1 1.12 MiB 0.10 MiB community/haskell-memory 0.13-2 4.22 MiB 0.39 MiB community/haskell-mime-types 0.1.0.7-2 15.68 MiB 0.92 MiB community/haskell-mmorph 1.0.6-2 1.37 MiB 0.11 MiB community/haskell-monad-control 184.108.40.206-2 1.02 MiB 0.08 MiB community/haskell-mtl 2.2.1-6 3.50 MiB 0.21 MiB community/haskell-network 220.127.116.11-1 6.67 MiB 0.58 MiB community/haskell-network-uri 18.104.22.168-3 5.30 MiB 0.32 MiB community/haskell-old-locale 22.214.171.124-8 1.10 MiB 0.07 MiB community/haskell-old-time 126.96.36.199-8 3.76 MiB 0.23 MiB community/haskell-pandoc-types 188.8.131.52-1 14.69 MiB 0.97 MiB community/haskell-parsec 3.1.11-2 7.63 MiB 0.57 MiB community/haskell-pcre-light 0.4.0.4-2 1.35 MiB 0.12 MiB community/haskell-pem 0.2.2-3 0.43 MiB 0.06 MiB community/haskell-primitive 0.6.1.0-2 4.29 MiB 0.20 MiB community/haskell-random 1.1-6 2.47 MiB 0.21 MiB community/haskell-resourcet 184.108.40.206-1 2.49 MiB 0.20 MiB community/haskell-scientific 0.3.4.9-1 2.42 MiB 0.26 MiB community/haskell-semigroups 0.18.2-3 0.27 MiB 0.04 MiB community/haskell-sha 220.127.116.11-5 2.25 MiB 0.18 MiB community/haskell-socks 0.5.5-5 3.84 MiB 0.34 MiB community/haskell-stm 18.104.22.168-3 1.32 MiB 0.13 MiB community/haskell-streaming-commons 0.1.15.5-3 4.65 MiB 0.40 MiB community/haskell-syb 0.6-4 5.07 MiB 0.25 MiB community/haskell-tagged 0.8.5-1 1.86 MiB 0.15 MiB community/haskell-tagsoup 0.14-1 17.53 MiB 1.29 MiB community/haskell-temporary 22.214.171.124-3 0.37 MiB 0.06 MiB community/haskell-texmath 0.8.6.5-3 90.46 MiB 6.30 MiB community/haskell-text 126.96.36.199-3 19.80 MiB 1.63 MiB community/haskell-tls 1.3.8-10 17.84 MiB 1.53 MiB community/haskell-transformers-base 0.4.4-5 0.62 MiB 0.06 MiB community/haskell-transformers-compat 0.5.1.4-3 0.29 MiB 0.04 MiB community/haskell-unix-compat 0.4.2.0-1 0.45 MiB 0.05 MiB community/haskell-unordered-containers 0.2.7.1-1 4.93 MiB 0.38 MiB community/haskell-utf8-string 188.8.131.52-2 1.80 MiB 0.18 MiB community/haskell-vector 0.11.0.0-6 46.14 MiB 2.20 MiB community/haskell-x509 1.6.3-18 6.18 MiB 0.52 MiB community/haskell-x509-store 1.6.1-18 0.42 MiB 0.06 MiB community/haskell-x509-system 1.6.3-15 0.38 MiB 0.06 MiB community/haskell-x509-validation 1.6.3-19 1.43 MiB 0.16 MiB community/haskell-xml 1.3.14-5 3.67 MiB 0.33 MiB community/haskell-yaml 0.8.18.4-1 19.86 MiB 2.13 MiB community/haskell-zip-archive 0.3.0.4-2 2.15 MiB 0.23 MiB community/haskell-zlib 0.6.1.1-3 3.79 MiB 0.31 MiB community/pandoc 1.17.2-9 50.47 MiB 6.28 MiB community/haskell-pandoc 1.17.2-9 135.20 MiB 10.19 MiB Total Download Size: 148.63 MiB Total Installed Size: 1959.78 MiB :: Proceed with installation? [Y/n]
As unless being stupid pandoc-rstudio has a conflicting issue with its dependency
==> pandoc-rstudio dependencies: - rstudio-desktop-bin (building from AUR) <SNIP SNIP> rstudio-desktop-bin: resolving dependencies... looking for conflicting packages... Package (5) New Version Net Change Download Size community/cmark 0.26.1-1 0.60 MiB 0.13 MiB community/pandoc 1.17.2-9 50.47 MiB 6.28 MiB community/pandoc-citeproc 0.10.1-9 46.59 MiB 5.59 MiB community/patchelf 0.9-1 0.14 MiB 0.04 MiB extra/r 3.3.1-2 53.62 MiB 33.98 MiB Total Download Size: 46.01 MiB Total Installed Size: 151.42 MiB <SNIP SNIP> pandoc-rstudio 1.13.1-1 0.00 MiB Total Installed Size: 0.00 MiB :: Proceed with installation? [Y/n] (1/1) checking keys in keyring [---------------------------------] 100% (1/1) checking package integrity [---------------------------------] 100% (1/1) loading package files [---------------------------------] 100% (1/1) checking for file conflicts [---------------------------------] 100% error: failed to commit transaction (conflicting files) pandoc-rstudio: /usr/bin/pandoc exists in filesystem pandoc-rstudio: /usr/bin/pandoc-citeproc exists in filesystem Errors occurred, no packages were upgraded.
Seeded this month: Arch 80 gig (and Talking~ 2, ~-anywhere 46, Apricity 16)
Desktop @3.3GHz 8 gig RAM, linux-ck
laptop1 Atom 2 gig RAM, Arch linux stock i686 (6H w/ 6yrs old battery ) & 2: ARM Tegra K1, 4 gig RAM, ChrOS
Atom Z520 2 gig RAM, OMV (Debian 7) kernel 3.16 bpo on SDHC | PGP Key: 0xFF0157D9
Community contribution threads are generally exempt from necro-bumping rules as most are not really dead, just dormant.
For non-Haskell users, building anything written in Haskell is a major PITA given the (sorry to say) insane dependencies that need to be pulled in (GHC alone is coming up on 1 GB). Luckily anything built with it can be statically compiled and reduced to relatively small binaries. If and when I get around to finishing the Xac rewrite, I will include the package in my binary repo for easy installation so most users can avoid the hassle of building it.
As a general update to this thread, I haven't touched the rewrite in nearly 3 years. It's not that I've given up on it. I just have other ongoing projects to work on and the rewrite itself requires a lot of time and effort to finish. For anyone curious, I wanted to enable full configuration of Xac sites via Lua, which required exposure of the Xac Haskell backend via Lua and the ability to call Lua functions from Haskell. I ended up writing what will eventually be a stand-alone project using template Haskell to automatically bind arbitrary Haskell data types and functions. Writing Haskell code in template Haskell to pick apart arbitrary data structures and move them on and off Lua's stack in order to create Lua data structures and vice-versa requires a lot of abstraction, concentration and time.
For anyone wondering wtf I would do that: Haskell is a fun and powerful language and I really like it. This was my pet project to dive into the deeper end of it and exposing the lower-level Pandoc code opens up a lot of interesting possibilities for Xac while drastically reducing overhead. I also really like Lua even though I've done nearly nothing with it. Using it as the configuration language introduces considerable versatility.
But yeah, vaporware for the moment.