Advantages of using Commvault WorkFlow

I created two important workflows on Commvault, they create and mount snapshot to hosts from EMC VMAX 3 and Hitachi VSP. They have been working without problems for months. So, I would like to talk about the advantages of using a Workflow.

First of all, the biggest advantage is the central management. Usually, snap scripts are distributed on different folders in different management servers. You can find them in “Task Scheduler” or “Crontab” if you are a little bit clever and lucky. But, it is a big deal in some environments. Also, the second deal is to find related files and figure out connections. If you use Commvault Workflow, all authorized employees can find and change scripts easily. And, to understand and see the big picture is very easy. You need a little time ( maybe minutes) to understand. On the other hand, in normal scripts, it can take hours, maybe days.

Secondly, security is very important every time. Normally, employees in “server team” can access your management server. Maybe, the other teams can also access your management server. So, they can access and change your script. But, if you use a workflow, only authorized employees can find and change scripts. On the other hand,  you can encrypt and hide passwords used in the scripts and even authorized employees can’t see passwords of important service users that are embedded in the script.

Thirdly, you don’t have to think about logging, mailing, catch errors, create alarms, etc. So you must only write pure codes. All the other stuff is ready. Normally, if you want to write a good script, you have to think about this stuff that most of the code consists of these. So, it’s hard to read and find something in the code like this. However, in Commvault Workflow, you can focus on pure code.

Finally, It can be integrated with CommVault features like IntelliSnap®, backup, restore, etc. Therefore, currently, the biggest problem is to create a consistent snap. And you can solve this problem easily with IntelliSnap® integrated scripts. Also, if you need periodically backup and restore scripts, you have all needs in the tool.

Consequently, Workflow in Commvault has many advantages and I only mentioned the most important ones here, but there are more.  As well, you can use ready-made scripts. You can find more information on the links below.

http://documentation.commvault.com/commvault/v11/article?p=features/workflow/workflow_overview.htm

https://ea.commvault.com/Education/LearningBytes/1002

http://documentation.commvault.com/commvault/v11/article?p=features/workflow/getting_started.htm

Commvault workflow design

Protect Commvault Disk Libraries from Ransomware

To protect disk libraries from Ramsomware, you can activate Ramsomware Protection on properties of Commvault Media agent

http://documentation.commvault.com/commvault/v11/article?p=features/disk_library/t_enable_ransomware_mediaagent.htm

Another important feature, you can get alert if commvault backed up encrypted ransomware files

http://documentation.commvault.com/commvault/v11/article?p=features/disk_library/r_ransomware_protection.htm


Security Tips for Storage and Backup Admins

1. EMC Networker: Change “remote access” option *@*  on every client” –> http://sysarticles.com/emc-networker-security-exploit-that-remote-access/

2. Veritas Netbackup: Remove No.Restrictions file –> http://sysarticles.com/security-expolit-on-netbackup-no-restrictions/

3. EMC VMAX: Change default password of SMC user in unisphere (take snapshot or backup before changing. If you would get “Failed to authenticate user.” error, there is no way to fix except restore or reinstall. And, If you reinstall unisphere you will loose performance database. There is a trick to not get this error.)

4. EMC VPLEX: Change default passwords of VPLEX. Default passwords of VPLEX is already complex, but anyone can find default passwords of service, admin and root users in the documents.

5. Commvault: Activate Ransomware Protection –> http://documentation.commvault.com/commvault/v11/article?p=features/disk_library/t_enable_ransomware_mediaagent.htm


Security expolit on Netbackup (No.Restrictions)

Sometimes we or some consultants put No.Restrictions file in “INSTALL DIR\netbackup\db\altnames” directory, this option  makes our work easier. But it is a security risk, because of every netbackup client on the your network can restore data from any other clients. So, anybody can restore data of your important servers, then they can lookup and/or copy your important datas. To close this security exploit, you must delete the file No.Restrictions.


EMC Networker security exploit that “Remote access”

If you didn’t care when you installed EMC networker software, some consultants enter an option *@* for every client, this means that every networker client on the your network can restore data from any other clients. So, anybody can restore data of your important servers, then they can lookup and/or copy your important datas.

You can check and change this option on “EMC Networker Administration –> Configuration –> Clients” then double click on client(or right click and select Modify Client Properties) and select Globals(2 of 2)

 

remote access       (read/write, string list)
              This  attribute  controls who may back up, browse, and recover a
              client's files.  By default this attribute  is  an  empty	 list,
              signifying that only users on the client are allowed to back up,
              browse, and recover its files.   Additional  users,  hosts,  and
              netgroups	 may  be  granted  permission  to access this client's
              files by adding their names to this attribute.   Netgroup	 names
              must  be	preceded by an ampersand ('&').	 Each line specifies a
              user  or	a  group  of  users,  using  one  of  these   formats:
              user/host@domain , group/host@domain , user@host , user@domain ,
              group@host , group@domain , &netgroup (only available  on	 plat-
              forms that support netgroups) , user_attribute=value[, ...].

              where  user is a user name; host is a host name; group is a user
              group name; domain is a domain name; user_attribute can be user,
              group,  host,  nwinstname, nwinstancename, domain, or domaintype
              (type of the domain, NIS or WINDOMAIN).

              The user attributes: nwinstname and nwinstancename are  used  to
              indicate	a  NetWorker  instance name.  The value that should be
              entered for either of these  attributes  is  the	value  in  the
              "name"  field  in	 the  NSRLA  resource  for the machine where a
              matched user is connecting from.

              value can be any string delimited by white space. If  the	 value
              has  space in it, then it can be quoted with double quotes.  The
              value may contain wild cards, "*".  Entering just	 a  user  name
              allows  that user to administer NetWorker from any host (equiva-
              lent to user@* or */user	or  user=user).	  Netgroup  names  are
              always preceded by an "&".

              The  format:  user_attribute=value[, ...] is more secure because
              the format is not overloaded. For example, if test@test.acme.com
              is entered, then any users in the test group or users named test
              and that are in the domain;  test.acme.com  or  from  the	 host;
              test.acme.com will match this entry.
              Example: The entries:

              remote access: mars, *@jupiter, sam@pluto, */root;

              remote  access:  host=mars, host=jupiter, "user=sam,host=pluto",
              user=root;

              are equivalent.

How to Expand a Striped Meta on EMC VMAX (Disk büyütme işlemi)

1. Create new meta devices that you will add to disk:

symconfigure -cmd “create dev count=X, size=cyl, emulation=FBA, config=TDEV, mvs_ssid=0, device_attr=SCSI3_persist_reserv;” prepare / commit

·        Change the X and Y to the correct values for your environment

·         Make a note of the device IDs, I’ll call them AAAA and BBBB, assuming you created two

Example: symconfigure -sid 096 -cmd “create dev count=8, size=27776 cyl, emulation=FBA, config=TDEV, mvs_ssid=0, device_attr=SCSI3_persist_reserv;” commit

New symdevs: 01D1C:01D23 [TDEVs]

 

2.      Create new BCV meta devices that exactly same as Striped Meta that you want expand :

symconfigure -cmd “create dev count=1, size=Y cyl, config=BCV+TDEV, emulation=FBA, mvs_ssid=0;” prepare / commit

·        Change the Y to the current size of the meta you are expanding

·         Note the device IDs, I’ll call them XXXX and YYYY, assuming auto meta settings created two devices

Example: symconfigure -sid 096 -cmd “create dev count=8, size=27776 cyl, config=BCV+TDEV, emulation=FBA, mvs_ssid=0;” commit

New symdevs:  01AAC-01AB3

 

3.       Create volume from BCV meta devices:

symconfigure -cmd “form meta from dev XXXX, config=striped, stripe_size=1 cyl; add dev YYYY to meta XXXX;” prepare / commit

you can make 2nd and 3th steps with unisphere:

a.        Select Storage -> volumes -> “Create meta volume” under BCV+TDEVS 

image001

b.        Select “Create volumes” and “Using New Virtual Volumes”.

image002

c.       Enter Member Count and Member Capacity that exacly same as stiraped meta that you want to expand , then select BCV+TDEV.

Number of Meta Volumes 1 * Meta Volume Capacity Meta Volume Member Count including Head Meta Volume Member Capacity 2776JJ Calculated Meta Volume Capacity 2034 GB ¡ 222 S Cyl * Volume Configuration B+TDEV

d.        Select “Run Now”, then copy device IDs.

 

4.      Bind this meta to a pool – symconfigure -cmd “bind tdev XXXX to Pool <POOL> preallocate size =ALL allocate_type = persistent;” prepare / commit

·         Replace <POOL> with one of your pool names

with unisphere:

a.        Select Bind under Storage -> Thin Pools -> Pool:

Rebalance Variance (1-50) 1 Maximum Volumes per Rebalance Scan .. 256 Pool Reserved Capacity Enabled ü Expand Bind

b.        Enter Volume ID, then select “Find Volumes” :

Thin Volunıes Wizard 1 Find Volumes Find volumes that match the following criteria Capacity equal to GB %. Volume ID 3335 (e.g. 001 or 001-OFF or 001 ,003-OTF) Volume Identifier Name Additional Criteria Select Category Add Another Clear All Find Volumes> Cancel Help

 

c.    Select related volume, “Allocate Full Volume Capacity” and “Persist preallocated capacity …” , then click Bind :

Selected 0 items Allocate Full Volume Capac) Persist preallocated capacity through reclaim or <Modify Criteria Bind Cancel Help

 

Not: if Dynamic RDF is enabled you will get  “ Error occurred while Defining change number 1:

   The devices being acted on are a mixture of dynamic and Non dynamic DRDF devices

   Device 1D1C generated the failure”

errors at 5th step. So you must enable Dynamic RDF on AAAA:BBBB devices

To enable Dynamic RDF :

symconfigure -sid aaa -cmd “set device AAAA:BBBB attribute=dyn_rdf;” commit

Example: symconfigure -sid 096 -cmd “set device 01D1C:01D23 attribute=dyn_rdf;”  commit

5.     Now you can add new meta devices:

symconfigure -cmd “add dev AAAA:BBBB to meta ZZZZ, protect_data=TRUE, bcv_meta_head=XXXX;” prepare / commit

·         AAAA and BBBB are the device IDs created in step 1

·         ZZZZ is the device IDs of the meta head you want to expand

·         XXXX is the device IDs of the BCV meta head created in step 3

·         Example: symconfigure -sid 096 -cmd “add dev 1D1C:1D23 to meta 08BC, protect_data=TRUE, bcv_meta_head=1AAC;” commit

 

6.    After expand operation you can unbind BCV volume, then dissolve and delete BCV meta devices. Look at: “EMC VMAX – Removal Of A TDEV”

 


Commvault Simpana 10 SP11 güncellemesi sırasında çıkan hata çözümü

Commvault Simpana 10 SP11 güncellemesi sırasında MSSQL servisleri kapandıktan sonra açılmıyor ise ve Event viewer Application Loglarında aşağıdaki hataları görüyor iseniz:

* Initializing the FallBack certificate failed with error code: 15, state: 1, error number: 6000.
* TDSSNIClient initialization failed with error 0x80092004, status code 0x80. Reason: Unable to initialize SSL support. Cannot find object or property.
* TDSSNIClient initialization failed with error 0x80092004, status code 0x1. Reason: Initialization failed with an infrastructure error. Check for previous errors. Cannot find object or property.
* SQL Server could not spawn FRunCM thread. Check the SQL Server error log and the Windows event logs for information about possible related problems.

 Çözüm:

Read More


Netbackup status code 239

Problem Tanımı: Günlerce, haftalarca hatta aylarca düzgün yedek aldıktan sonra birden bire aşağıdaki hata kodunu ve ekran görüntüsünü almaya başladıysanız, bu çözüm sizin için:

Hata Kodu: 239

Ekran Görüntüsü:

 hatakodu239

Ana job:
4/19/2014 11:43:48 PM - Info nbjm(pid=5772) starting backup job (jobid=1627586) for client clientname, policy MSSQL_clientname_DBNAME_test, schedule Full_1M 4/19/2014 11:43:48 PM - Info nbjm(pid=5772) requesting MEDIA_SERVER_WITH_ATTRIBUTES resources from RB for backup job (jobid=1627586, request id:{30E2C633-C9C4-4520-9F0B-F291957FDA12})
4/19/2014 11:43:48 PM - requesting resource stu_disk_sp-nbappliance21
4/19/2014 11:43:48 PM - requesting resource masterserver.domain.com.NBU_CLIENT.MAXJOBS.clientname
4/19/2014 11:43:48 PM - requesting resource masterserver.domain.com.NBU_POLICY.MAXJOBS.MSSQL_clientname_DBNAME_test
4/19/2014 11:43:48 PM - granted resource masterserver.domain.com.NBU_CLIENT.MAXJOBS.clientname
4/19/2014 11:43:48 PM - granted resource masterserver.domain.com.NBU_POLICY.MAXJOBS.MSSQL_clientname_DBNAME_test
4/19/2014 11:43:48 PM - granted resource stu_disk_sp-nbappliance21
4/19/2014 11:43:48 PM - estimated 0 Kbytes needed
4/19/2014 11:43:48 PM - Info nbjm(pid=5772) started backup (backupid=clientname_1397940228) job for client clientname, policy MSSQL_clientname_DBNAME_test, schedule Full_1M on storage unit stu_disk_sp-nbappliance21
4/19/2014 11:43:49 PM - started process bpbrm (9628)
4/19/2014 11:43:54 PM - connecting
4/19/2014 11:43:54 PM - Info bpbrm(pid=9628) clientname is the host to backup data from    
4/19/2014 11:43:54 PM - Info bpbrm(pid=9628) reading file list for client       
4/19/2014 11:43:55 PM - Info bpbrm(pid=9628) starting bphdb on client        
4/19/2014 11:43:55 PM - Info bphdb(pid=15948) Backup started          
4/19/2014 11:43:55 PM - connected; connect time: 0:00:01
4/19/2014 11:43:57 PM - Info dbclient(pid=17552) INF - BACKUP STARTED USING       
4/19/2014 11:43:57 PM - Info dbclient(pid=17552) Microsoft SQL Server 2012 (SP1) - 11.0.3381.0 (X64)    
4/19/2014 11:43:57 PM - Info dbclient(pid=17552) Aug 23 2013 20:08:13        
4/19/2014 11:43:57 PM - Info dbclient(pid=17552) Copyright (c) Microsoft Corporation        
4/19/2014 11:43:57 PM - Info dbclient(pid=17552) Enterprise Edition: Core-based Licensing (64-bit) on Windows NT 6.2 <X64> (Build 9200: )
4/19/2014 11:43:57 PM - Info dbclient(pid=17552) Batch = C:\Nbu_Scripts\full_dbname.bch, Op# = 1      
4/19/2014 11:43:57 PM - Info dbclient(pid=17552) INF - Using backup images SP-clientNODEB.MSSQL7.clientname.db.dbname.~.7.001of008.20140419234356..C to SP-clientNODEB.MSSQL7.clientname.db.North008of008.20140419234356.C    
4/19/2014 11:43:58 PM - Info dbclient(pid=17552) INF - backup database "dbname" to VIRTUAL_DEVICE='VNBU0-17552-15940-1397940237', VIRTUAL_DEVICE='VNBU1-17552-15940-1397940237', VIRTUAL_DEVICE='VNBU2-17552-15940-1397940237', VIRTUAL_DEVICE='VNBU3-17552-15940-1397940237', VIRTUAL_DEVICE='VNBU4-17552-15940-1397940237', VIRTUAL_DEVICE='VNBU5-17552-15940-1397940237', VIRTUAL_DEVICE='VNBU6-17552-15940-1397940237', VIRTUAL_DEVICE='VNBU7-17552-15940-1397940237' with  stats = 10, blocksize = 65536, maxtransfersize = 65536, buffercount = 16
4/19/2014 11:43:58 PM - Info dbclient(pid=17552) INF - Number of stripes: 8, Number of buffers per stripe 2.
4/19/2014 11:43:58 PM - Info dbclient(pid=17552) INF - Created VDI object for SQL Server instance <clientname.anadolu.com>. Connection timeout is <300> seconds.
4/19/2014 11:44:17 PM - Info dbclient(pid=17552) ERR - Error in VxBSACreateObject: 3.      
4/19/2014 11:44:17 PM - Info dbclient(pid=17552)     CONTINUATION: - System detected error, operation aborted. 
4/19/2014 11:44:17 PM - Info dbclient(pid=17552) ERR - Error in GetCommand: 0x80770004.      
4/19/2014 11:44:17 PM - Info dbclient(pid=17552) ERR - Error in GetCommand: 0x80770004.      
4/19/2014 11:44:17 PM - Info dbclient(pid=17552) ERR - Error in GetCommand: 0x80770004.      
4/19/2014 11:44:17 PM - Info dbclient(pid=17552) ERR - Error in GetCommand: 0x80770004.      
4/19/2014 11:44:17 PM - Info dbclient(pid=17552) ERR - Error in GetCommand: 0x80770004.      
4/19/2014 11:44:17 PM - Info dbclient(pid=17552) ERR - Error in GetCommand: 0x80770004.      
4/19/2014 11:44:17 PM - Info dbclient(pid=17552) ERR - Error in GetCommand: 0x80770004.      
4/19/2014 11:44:17 PM - Info dbclient(pid=17552) ERR - Error in GetCommand: 0x80770004.      
4/19/2014 11:44:17 PM - Info dbclient(pid=17552)     CONTINUATION: - An abort request is preventing anything except termination actions.
4/19/2014 11:44:17 PM - Info dbclient(pid=17552)     CONTINUATION: - An abort request is preventing anything except termination actions.
4/19/2014 11:44:17 PM - Info dbclient(pid=17552)     CONTINUATION: - An abort request is preventing anything except termination actions.
4/19/2014 11:44:17 PM - Info dbclient(pid=17552)     CONTINUATION: - An abort request is preventing anything except termination actions.
4/19/2014 11:44:17 PM - Info dbclient(pid=17552)     CONTINUATION: - An abort request is preventing anything except termination actions.
4/19/2014 11:44:17 PM - Info dbclient(pid=17552)     CONTINUATION: - An abort request is preventing anything except termination actions.
4/19/2014 11:44:17 PM - Info dbclient(pid=17552)     CONTINUATION: - An abort request is preventing anything except termination actions.
4/19/2014 11:44:17 PM - Info dbclient(pid=17552)     CONTINUATION: - An abort request is preventing anything except termination actions.
4/19/2014 11:44:17 PM - Info dbclient(pid=17552) ERR - Error in VxBSACreateObject: 3.      
4/19/2014 11:44:18 PM - Info dbclient(pid=17552)     CONTINUATION: - System detected error, operation aborted. 
4/19/2014 11:44:19 PM - Info dbclient(pid=17552) ERR - Error in VxBSACreateObject: 3.      
4/19/2014 11:44:19 PM - Info dbclient(pid=17552)     CONTINUATION: - System detected error, operation aborted. 
4/19/2014 11:44:20 PM - Info dbclient(pid=17552) ERR - Error in VxBSACreateObject: 3.      
4/19/2014 11:44:20 PM - Info dbclient(pid=17552)     CONTINUATION: - System detected error, operation aborted. 
4/19/2014 11:44:20 PM - Info dbclient(pid=17552) INF - OPERATION #1 of batch C:\Nbu_Scripts\full_dbname.bch FAILED with STATUS 1 (0 is normal). Elapsed time = 24(24) seconds.
4/19/2014 11:44:21 PM - Info dbclient(pid=17552) ERR - Error in VxBSACreateObject: 3.      
4/19/2014 11:44:21 PM - Info dbclient(pid=17552)     CONTINUATION: - System detected error, operation aborted. 
4/19/2014 11:44:22 PM - Info dbclient(pid=17552) ERR - Error in VxBSACreateObject: 3.      
4/19/2014 11:44:22 PM - Info dbclient(pid=17552)     CONTINUATION: - System detected error, operation aborted. 
4/19/2014 11:44:22 PM - Info dbclient(pid=17552) INF - Results of executing <C:\Nbu_Scripts\full_dbname.bch>:      
4/19/2014 11:44:22 PM - Info dbclient(pid=17552) <0> operations succeeded. <1> operations failed.      
4/19/2014 11:44:22 PM - Info dbclient(pid=17552) INF - The following object(s) were not backed up successfully.  
4/19/2014 11:44:22 PM - Info dbclient(pid=17552) INF - dbname         
4/19/2014 11:44:22 PM - Error bpbrm(pid=9628) from client clientname: ERR - command failed: none of the requested files were backed up (2)
4/19/2014 11:44:22 PM - Error bpbrm(pid=9628) from client clientname: ERR - bphdb exit status = 2: none of the requested files were backed up
4/19/2014 11:44:27 PM - Info bphdb(pid=15948) done. status: 2: none of the requested files were backed up 
4/19/2014 11:44:27 PM - end writing
none of the requested files were backed up(2)


Child job:


 hatakodu239_child_job

Çözüm:

MSSQL yedeklerinde kullandığınız script içerisindeki sunucu adı ile politika içerisindeki sunucu adı birebir aynı olmasını sağlayın. Birinin sonunda domain adı var diğerinde yok ise eksik olana domain adını ekleyin. Biri küçük harf, diğeri büyük harfler ile yazılmış ise ikisininde karakter büyüklüklerini birebir aynı yapın. Sorun düzelecektir.

 


“Policy Storage” değiştirildikten sonra eski “storage unit” ten yedek almaya devam etme problemi

Netbackup Version: 7.6.0.1

Hata Tanımı: Politika içerisinden “policy storage” kısmındaki yedekleme yapılacak yeri değiştirmemize rağmen yedeği eski yere almaya devam etmesi.

Geçici Çözüm: Politikaya sağ tıklayarak “Copy to New Policy” seçeneğini seçmek, yeni politika üzerinden yedeklemeye devam etmek.

Kalıcı çözüm: Her zaman olduğu gibi, muhtemelen versiyon güncelleme gerekmektedir 🙂 Bulanlar bana e-posta atar ise burda yayımlarım.


Netbackup ‘ da senelerdir uğraştıran MSSQL yedeğinde yarıda kesilme problemi ve çözümü

Netbackup kullanmaya başladım başlayalı bir yarıda kesilme problemidir, gidiyordu. Açılan caseler, yapılan çalışmalar fayda vermiyordu ki en son yurt dışından gelen mühendis kök sebebi tam olarak bulamasada bir çözüme kavuşturdu. Ben çektim, siz çekmeyin. Buyrun çözüm:

Read More