May 18, 2018

Replacing Google Drive with Nextcloud

In an effort to increase privacy and move away from externally hosted solutions, I decided to move all my files from Google Drive to Nextcloud. I checked out various alternatives, including owncloud and seafile, but Nextcloud seemed the simplest from an end user interface perspective. Also, clients for mobile devices behaved similarly enough to Google apps that my wife was happy. All right, let's get started:

Install Prerequisites

sudo apt install -y apache2 mariadb-server libapache2-mod-php7.0 \ php7.0-gd php7.0-json php7.0-mysql php7.0-curl php7.0-mbstring \ php7.0-intl php7.0-mcrypt php-imagick php7.0-xml php7.0-zip

Setup the SQL database

We installed MariaDB in the previous step. Run the command as normal user who will access database.
WARNING: if you run below as root, sql will only accept the password as root.
mysql_secure_installation

If you ignored my warning and ran as root:

sudo mysql -u root -p
use mysql;
update user set plugin='' where User='root';
flush privileges;
\q

Login to SQL

mysql -u root -p

Create a new database for nextcloud ...with filename emoji support

Change database, username and password to match what you want:

CREATE DATABASE nextcloud CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
CREATE USER 'USERNAME'@'localhost' IDENTIFIED BY 'PASSWORD';

Assign privileges to user

NOTE: Without the grant option, setup will fail

GRANT ALL PRIVILEGES ON *.* TO 'USERNAME'@'localhost' WITH GRANT OPTION;
FLUSH PRIVILEGES;
QUIT;

Modify SQL format to support Nextcloud

Add following to:
sudo nano /etc/mysql/conf.d/my.cnf

[mysqld]
innodb_large_prefix=on
innodb_file_format=barracuda
innodb_file_per_table=true

Installing NextCloud to web server

v12 was current at time of writing, check Nextcloud for latest.
wget https://download.nextcloud.com/server/releases/nextcloud-12.0.3.tar.bz2

Extract files

tar -xjf nextcloud-*.tar.bz2

Move to web root

sudo cp -r nextcloud /var/www

Set file permissions

sudo chown -R www-data:www-data /var/www/nextcloud/

Apache webserver config

Test, then disable default site

Confirm Apache default site loads using http://HOSTNAME then disable it using the below:
sudo a2dissite 000-default
sudo service apache2 reload

Create a config for Nextcloud

sudo nano /etc/apache2/sites-available/nextcloud.conf

Paste the following:

Alias / "/var/www/nextcloud/"

<Directory /var/www/nextcloud/>
  Options +FollowSymlinks
  AllowOverride All

 <IfModule mod_dav.c>
  Dav off
 </IfModule>

 SetEnv HOME /var/www/nextcloud
 SetEnv HTTP_HOME /var/www/nextcloud

</Directory>

Create a system link to enable nextcloud

Create a symlink to /etc/apache2/sites-enabled:
sudo ln -s /etc/apache2/sites-available/nextcloud.conf /etc/apache2/sites-enabled/nextcloud.conf

Enable Apache modules

sudo a2enmod rewrite headers env dir mime

Globally set servername

sudo nano /etc/apache2/apache2.conf
at bottom add:
ServerName server_domain_or_IP

Check config before breaking everything

sudo apache2ctl configtest

Restart Apache to apply

sudo service apache2 restart

Firewall the webserver

It's generally a bad idea to let a publically exposed server run without a firewall. We'll enable UFW to protect us from malicious actors. If you haven't allowed SSH and are remotely connected, you will be locked out.
sudo ufw allow in "Apache Full"
sudo ufw enable

Notes

Fix possible caldav errors

If your Nextcloud instance is installed in a subfolder called nextcloud and you’re running Apache create or edit the .htaccess file within the document root of your Web server and add the following lines:
sudo nano /var/www/.htaccess

Redirect 301 /.well-known/carddav /nextcloud/remote.php/dav
Redirect 301 /.well-known/caldav /nextcloud/remote.php/dav

Cloudflare

Cloudflare is awesome but if you minify JS, it may break things like the login button or user creation. Disable it and see.

To restore a database from backup

If you break everything and need to restore from a backup (you do have a backup, right?)

Destroy the exisiting database:

mysql -h [server] -u [username] -p[password] -e "DROP DATABASE nextcloud"

Create a new one:

mysql -h [server] -u [username] -p[password] -e "CREATE DATABASE nextcloud CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci"

Restore the backup:

mysql -h [server] -u [username] -p[password] [db_name] < nextcloud-sqlbkp.bak

Put Nextcloud in maintenance mode

sudo -u www-data php /var/www/nextcloud/occ maintenance:mode --on

Update data fingerprint or clients will fail

sudo -u www-data php /var/www/nextcloud/occ maintenance:data-fingerprint

Pretty URLs

Pretty URLs remove the index.php-part in all Nextcloud URLs, for example in sharing links like https://example.org/nextcloud/index.php/s/Sv1b7krAUqmF8QQ, making URLs shorter and thus prettier.

mod_env and mod_rewrite must be installed on your webserver and the .htaccess must be writable by the HTTP user. Then you can set in the config.php two variables:

'overwrite.cli.url' => 'https://example.org/nextcloud',
'htaccess.RewriteBase' => '/nextcloud',

Example, if your setup is available on https://example.org/nextcloud or:

'overwrite.cli.url' => 'https://example.org',
'htaccess.RewriteBase' => '/',

If it isn’t installed in a subfolder. Finally run this occ-command to update your .htaccess file:
sudo -u www-data php /var/www/nextcloud/occ maintenance:update:htaccess

After each update, these changes are automatically applied to the .htaccess file.

SSL

NOTE: Have to do this before before pretty urls

sudo a2enmod ssl
sudo a2ensite default-ssl
sudo service apache2 reload

let's encrypt

sudo apt update && \
sudo apt install software-properties-common && \
sudo add-apt-repository ppa:certbot/certbot && \
sudo apt update && \
sudo apt install -y python-certbot-apache 

configure:

sudo certbot --apache
sudo certbot renew --dry-run
sudo crontab -e
add:
0 0 5 * * certbot renew

Backups

/etc/letsencrypt/
/etc/apache2/sites-available/
/var/www/nextcloud/
and whatever path your data directory is set to

enable HSTS

sudo nano /etc/apache2/sites-available/000-default-le-ssl.conf

add:

<IfModule mod_headers.c>
  Header always set Strict-Transport-Security "max-age=15552000; includeSubDomains"
</IfModule>

sudo service apache2 restart

server tuning

https://docs.nextcloud.com/server/12/admin_manual/configuration_server/server_tuning.html

caching

local caching

we dont use apcu anymore, I had a reason, forget why

sudo apt install php-apcu
After restarting your Web server, add this line to your config.php file:
'memcache.local' => '\OC\Memcache\APCu',

remote cache and file locks

sudo apt install redis-server php-redis
sudo nano /etc/redis/redis.conf

Now, find and change:
port 6379
to
port 0

Then uncomment:

unixsocket /var/run/redis/redis.sock
unixsocketperm 700 changing permissions to 770 at the same time: unixsocketperm 770

Save and quit, then add the Redis user redis to the www-data group:
sudo usermod -a -G redis www-data

Finally, restart Apache with:
sudo service apache2 restart

And start Redis server with:
sudo service redis-server start

Restart Apache, edit config.php, refresh the Nextcloud admin page. This example config.php configuration uses Redis for the local server cache:

'memcache.local' => '\OC\Memcache\Redis',
'redis' => array(
     'host' => 'localhost',
     'port' => 6379,
      ),

For best performance, use Redis for file locking by adding this:
'memcache.locking' => '\OC\Memcache\Redis',
If you want to connect to Redis configured to listen on an Unix socket (which is recommended if Redis is running on the same system as Nextcloud) use this example config.php configuration:

'memcache.local' => '\OC\Memcache\Redis',
'redis' => array(
     'host' => '/var/run/redis/redis.sock',
     'port' => 0,
     'dbindex' => 0,
     'password' => 'secret',
     'timeout' => 1.5,
      ),

Only host and port variables are required, the other ones are optional.

php.ini file changes

sudo nano /etc/php/7.0/apache2/php.ini

opcache.enable=1
opcache.enable_cli=1
opcache.interned_strings_buffer=8
opcache.max_accelerated_files=10000
opcache.memory_consumption=128
opcache.save_comments=1
opcache.revalidate_freq=1

php.ini changes that need to be made in webroot .htaccess

sudo nano /var/www/nextcloud/.htaccess
php_value memory_limit 512M
sudo systemctl restart apache2.service

enable free docs editing?

not yet setup

https://nextcloud.com/collaboraonline/
https://serenity-networks.com/how-to-install-onlyoffice-document-server-for-nextcloud-fast-easy/

mounting smb shares

we do this using an nfs mount now, much faster

sudo apt install smbclient php-smbclient
mounting will fail if server has min protocol value set in smb.conf
instead set:

protocol = smb3 #win8 or higher clients

audio player

add external storage music nfs/smb shares, then:

cd /var/www/nextcloud/ && \
sudo -u www-data php occ audioplayer:scan --all

large size uploads

https://docs.nextcloud.com/server/12/admin_manual/configuration_files/big_file_upload_configuration.html

cron job

To run a cron job on a *nix system, every 15 minutes, under the default Web server user (often, www-data or wwwrun), you must set up the following cron job to call the cron.php script:
sudo crontab -u www-data -e

add:

*/15  *  *  *  * php -f /var/www/nextcloud/cron.php

You can verify if the cron job has been added and scheduled by executing:
crontab -u www-data -l

*/15  *  *  *  * php -f /var/www/nextcloud/cron.php

above should appear if all good

backup

backup the database:

mysqldump --single-transaction -h [server] -u [username] -p[password] [db_name] > nextcloud-sqlbkp_`date +"%Y%m%d"`.bak
sudo mysqldump --defaults-extra-file="/tank/sqlbak/.sqlpwd" --single-transaction -h localhost -u nextcloud nextcloud > /tank/sqlbak/nextcloud-sqlbkp_`date +"%Y%m%d"`.bak

setup password file

sudo nano /tank/sqlbak/.sqlpwd sudo chmod 600 /tank/sqlbak/.sqlpwd && sudo chown root:nogroup /tank/sqlbak/.sqlpwd

contents

password must be in quotes

.sqlpwd contents
[mysqldump] user = DATABASEUSERNAME password = "DATABASEPASSWORD"

setup cron job for db backup

this doesnt work try below

0 0 * * * OUT=$(mysqldump --defaults-extra-file="/pathto/sqlbak/.sqlpwd" --single-transaction -h localhost -u USERNAME nextcloud > /pathto/sqlbak/nextcloud-sqlbkp_`date +"%Y%m%d"`.bak) || echo "$OUT"

below

0 0 * * * OUT=$(mysqldump --defaults-extra-file="/pathto/sqlbak/.sqlpwd" --single-transaction -h localhost -u USERNAME nextcloud > /pathto/sqlbak/nextcloud-sqlbkp_`date +"\%Y\%m\%d"`.bak) || echo "$OUT"

backup the data (we use duplicacy for this instead)

rsync -Aax nextcloud/ nextcloud-dirbkp_`date +"%Y%m%d"`/

cronjob for audio scan

#!/bin/bash
cd /var/www/nextcloud
sudo -u www-data php occ audioplayer:scan --all

If you need to clear the audioplayer index:
sudo -u www-data php occ audioplayer:reset --all

cronjob for preview image scan

https://github.com/rullzer/previewgenerator
Add a (system) cron job for ./occ preview:pre-generate

Run every 10 minutes

#script
#!/bin/bash
cd /var/www/nextcloud
sudo -u www-data php occ preview:pre-generate

music app (instead of audio player)

Reset scanned metadata

Reset all data stored to the music database. Target either specified user(s) or all users.

Warning: This command will erase user-created data! It will remove all tracks from playlists as playlists are linked against the track metadata.

./occ music:reset-database USERNAME1 USERNAME2 ...
./occ music:reset-database --all
./occ music:reset-cache --all

then install from apps
then configure
then scan
NOTE: this music player doesnt work, it just chokes on the library

Two-factor Authentication

If a two-factor provider app is enabled, it is enabled for all users by default (though the provider can decide whether or not the user has to pass the challenge). In the case of an user losing access to the second factor (e.g. lost phone with two-factor SMS verification), the admin can temporarily disable the two-factor check for that user via the occ command:
sudo -u www-data php occ twofactor:disable <username>

To re-enable two-factor auth again use the following commmand:
sudo -u www-data php occ twofactor:enable <username>

preview generator

install app into nextcloud
then run:
cd /var/www/nextcloud
sudo -u www-data php occ preview:pre-generate
sudo chown root /usr/local/bin/cloud-preview-gen.sh
sudo chmod 600 /usr/local/bin/cloud-preview-gen.sh
sudo chmod +x /usr/local/bin/cloud-preview-gen.sh

sudo crontab -e -u www-data
cron job:

46 1 * * * /usr/local/bin/cloud-preview-gen.sh

Script for cronjob:

#!/bin/bash
cd /var/www/nextcloud
sudo -u www-data php occ preview:pre-generate

https://www.scalingphpbook.com/blog/2014/02/14/best-zend-opcache-settings.html

Backup Folders

Simply copy your config, data and theme folders (or even your whole Nextcloud install and data folder) to a place outside of your Nextcloud environment. You could use this command:

rsync -Aax nextcloud/ nextcloud-dirbkp_`date +"%Y%m%d"`/

NOTE: need to automate this!

increasing RNG

sudo apt install haveged

make sure start on boot

sudo update-rc.d haveged defaults

install rng tools for testing

sudo apt install rng-tools

test

See how quick this comes back before and after haveged installed
cat /dev/random | rngtest -c 1000

increasing PHP limits

this breaks integrity checks
where do we edit this?
not in user.ini either
sudo nano /var/www/nextcloud/.user.ini upload_max_filesize = 2048M post_max_size = 2058M post max needs to be bigger by a bit

if you break the htaccess file do this:

sudo -u www-data php /var/www/nextcloud/occ maintenance:update:htaccess

if apcu fails/not available after install

sudo nano /etc/php/7.0/apache2/conf.d/20-apcu.ini

apc.enabled=1
apc.enable_cli=1
apc.shm_size=128M
apc.ttl=3600
apc.user_ttl=7200
apc.gc_ttl=3600

or edit php.ini and add extension=apcu.so

icon generation:

Nextcloud will automatically generate favicons and home screen icons depending on the current app and theming color.

This requires the following additional dependencies:

    PHP module imagick
    SVG support for imagick (e.g. libmagickcore5-extra)

sudo apt install libmagickcore-6.q16-2-extra -y