We are so grateful to the people that contribute to our project. Recently thanks to Andrew Mind there is a LizardFS manual available in Russian.
Next contributions are coming up shortly so stay tuned!
It is a general trend which is getting stronger and stronger. Moving away from closed-source, proprietary solutions to COTS, open infrastructure, solutions that enable people to use commodity hardware.
You could see that in a couple of talks at #NABshow. People are not only working on them, some of them are already using such solutions.
Embrace the shift from #fiberchannel to #ethernet Find out how you can benefit from #opensource hardware agnostic #scaleout high-performance #distributed #filesystem Visit #lizardfs booth on #NAB2018 N2936sul-b (North Hall).
As everything that we do Hadoop Plugin for LizardFS is as simple as we could make it.
This is a java based solution allowing Hadoop to use LizardFS storage, implementing an HDFS interface to LizardFS. It functions as kind of a File System Abstraction Layer. It enables you to use Hadoop jobs to directly access the data on a LizardFS cluster. The plugin translates LizardFS protocol and makes the metadata readable for Yarn and Map Reduce. For performance, Hadoop nodes should run on the same machines as LizardFS chunk servers.
LizardFS mount gives direct access to stored files, from the OS level. This allows you to use it as a shared storage in your company and a computation storage for HADOOP at the same time. It is not required to use HADOOP tools to put/get files from your storage in comparison to HDFS. We can also take advantage of Erasure Coding and save a lot of disk space (HDFS recommends to store 3 copies).
public BlockLocation getFileBlockLocations(FileStatus file, long start, long len)
Returns information where data blocks are held in your LizardFS installation. If Hadoop is run on the same machines, it can take advantage of data locality.
To install Hadoop with LizardFS:
1) Install and setup LizardFS cluster
2) Install HADOOP – but don’t start
3) Install LizardFS-HADOOP plugin on all HADOOP nodes
4) Configure LizardFS-Plugin in HADOOP (alongside HDFS or replace it)
5) Start HADOOP
Let us know what you think of it.
Come visit our stand at NABSHOW in Las Vegas.
North Hall Central Lobby in the Startup Loft, Booth number: N2936SUL-B
After so many tests we decided to release pre-alfa, cutting-edge Hadoop connector for LizardFS.
You can download it from here.
We are waiting for the feedback.
At the moment you will be required to build the binaries yourself.
We are looking forward to some feedback. Please bear in mind that we are not Hadoop experts, thu we might have missed some test scenarios.
We really need a help from the community site on this one. Help greatly appreciated and needed.
Let us know if you will be at #MWC18
Would love to meet up and discuss storage
Come visit us at our stand and see how #LizardFS can help you!
Find us in the Innovation Zone at Hollywood Professionals Association Tech Retreat.
As promised for some time now, we finally released our High Availability mechanism to our open source community. Enjoy and let us know what you think!
There are many ways to access Distributed File System. For us the most favorite way is through native clients – does not matter whether it is Linux, Mac or Windows.
But what when you cannot install third party software on your client?
Or you need storage for systems that there is no client for?
NFS might be the answer for you. The simplest way to use it would be to create server/gateway. That solution has obvious drawbacks (lack of HA, performance, poor scalability).
We knew that we can do better.
So we did.
How does it work?
Let’s start from NFS 3.x
On each chunk server, there is an NFS server which enables clients to connect to the LizardFS cluster and read/write files via the NFS protocol. Now you can use LizardFS to create a Network Attached Storage solution nearly out of the box. It doesn’t matter what system you are running, as long as it supports NFS you can mount up to 1 Exabyte of storage from a LizardFS cluster to your machine.
Some demanding users might immediately ask questions like: ok but what about chunkserver failure?
Well if you are that exigent you will not mind discussing a support contract with us to not only get peace of mind but also a truly highly available solution.
What about NFS 4.1 and pNFS?
The story is just getting more and more interesting here. Now LizardFS is not only supporting NFS 4.0 but also provides parallel reads and writes through the parallel Network File System (pNFS) plus you are getting High Availability as a bonus. Extra add-ons do not end here. Thanks to NFS4.x support you can use Kerberos authentication for the clients.
Obvious use cases of NFS support
RedHat Enterprise Linux > 6.4
SuSE Linux Enterprise Server > 11 sp. 3
We are going to test various virtualisation solutions and see how is performing with them. Obvious differences should be observed in the solutions that are already capable of using pNFS like:
- Redhat Virtualization
- KVM on modern Linux systems
- XEN on modern Linux Systems
We are also interested in seeing the results of tests with others:
- VMware (Although vSphere 6 includes NFSv4.1 support, it does not include pNFS! I)
- Citrix XenServer (no pNFS)
- HyperV (no pNFS)
- Windows Server (no pNFS)
- Windows (no pNFS)
Although NFS seems to be one of the most popular protocols different solutions are supporting different versions of it. It has certain consequences. For instance, with NFS 3 there is no direct support for ACLs. No parallelism in that version has also substantial impact on the performance.
So while having a unified environment in regards to communication protocols sounds really good, you need to first analyze what OSs are running on your infrastructure before making the final decision of going that way.
Fortunately most of the times we have an option of using other protocols like SMB or once it is acceptable to install additional software on a client machine go with the option of native clients.
Key differentiators between NFS versions
NFS3 – stateless protocol, supports only UNIX semantics, weak security, identification via UID/GID, no delegations.
NFS4 – stateful protocol, UNIX and Windows semantics, strong authentication via kerberos, string based identification (user@host…), delegations possible
pNFS – all the adavantages of NFS4 plus parallelised access to resources
Which platforms support what versions and features of NFS
|Platform||Version||NFS version||pNFS support||NFS broken||Comments|
|RedHat||6.3||4.1||native||up to 6.5 problems with NFS in general on RH||Read more|
|SuSE SLES||11 sp. 3||4.1||native|
|Linux Kernel||2.6.39||4.1||native||requires the proper version of nfs-utils to work|
|Ubuntu||14.04||4.1||native||some broken support from 12.04||VMWare seems to have problems implementing proper NFS support for ages now|
|VMWare||6.5||4.1||none||pNFS not implemented|
|Citrix XenServer||7||4.1||none||pNFS not implemented||See bug|
|Proxmox||4||4.1||native||Based on Debian 9, so full support for pNFS|
|Redhat Virtualization Server||Native pNFS support if based on RHEL > 6.5|
|XEN||Depends on OS, works on RHEL/Derivative > 6.4 and SLES > 11.3 and Debian >8 and Ubuntu >= 14.04. Not sure which others.|
|Oracle VM||3.4||native||If running on RHEL/Oracle Linux > 6.4|
|Windows Server||2016||3||none||Windows only supports NFS v3|
|Solaris||11||4||none||There was a prototype made available a few years ago when OpenSolaris was still alive, as of today, Solaris has no support for pNFS.|
|AIX||6||4||none||pNFS not implemented|
|Amazon EFS||4.1||none||pNFS not implemented|
|Oracle dNFS||12CR2||4.1||native||some minor problems that limit full performance, but still faster than NFSv3||Oracle has a NFS implementation inside its RDBMS. It support pNFS from 12Cr2. The support is from 2017 and still has some little quirks.|
|OpenStack||Icehouse||4.1||native||pNFS not implemented||As of the Icehouse release, the NFS driver (and other drivers based off it) will attempt to mount shares using version 4.1 of the NFS protocol (including pNFS).|
SIGN UP FOR OUR NEWSLETTER
Stay in the know for news and updates. Subscribe now!
We have a clear goal which we have set ourselves - to deliver the best Software Defined Storage solution. However, we know even when we reach our current objective, we will not stop innovating. Check out the new version of LizardFS and judge for yourself.
1691 Kettering Street
Irvine, CA, 92614
Tel: (+1) 646 655 0 693
Tel: (+48) 22 100 32 48
Officially part of: